Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complementary ability to satisfy negated constraints, for example, generating "a non-red cup." Measuring negation raises challenges not faced by affirmation-based benchmarks and requires careful prompt and evaluation design. We introduce NegT2IBench, a benchmark of 4,800 prompts covering two attribute types and four relation categories. Prompts are organized by polarity: the number of positive statements that must hold and negated statements that must not, each ranging from 0 to 2. Varying the two independently separates the effect of negation from the effect of prompt complexity. Our detector-based scoring is reproducible, auditable, and pinpoints which requirement failed. On 600 images with three-annotator labels, it agrees with humans as closely as vision-language judges up to 30x larger, while using only a fraction of their GPU memory. Across eleven T2I models and 211,200 images, nine score lower on a single negated statement than on a single positive one. Per-statement scoring reveals that the loss is largest for color and near zero for proximity, and that 41.5% of failed statements render exactly what the prompt forbids. Rendering what a prompt asks for and withholding what it forbids are distinct capabilities that an aggregate compositional score cannot distinguish. NegT2IBench measures the latter directly, providing a controlled testbed for diagnosing negation failures and developing methods to overcome them.
Figures & tables
Figure 1: NegT2IBench prompts mix positive (blue) and negated (pink) statements, spanning six categories (above image, right) and organized by the number of each statement (above image, left). Images come from the benchmarked T2I models. The box under each image gives the human verdict; marks show whether each evaluator agrees (✓) or disagrees (✗) with it.
Category
Values
Positive example
Negated example
Attributes
Color (10)
red, orange, yellow, green, blue, purple, pink, brown, black, white
a red cup
a non-red cup
Material (4)
wooden, metal, plastic, glass
a glass cup
a non-glass cup
Relations
Directional (4)
left of, right of, above, below
the backpack is above the parking meter
the backpack is not above the parking meter
Proximity (2)
near, far from
the backpack is far from the parking meter
the backpack is not far from the parking meter
Table 1: NegT2IBench categories. Each category has 800 prompts (100 prompts per polarity setting, 8 settings, 4,800 prompts over 79 object classes). Each row shows a matched pair from the benchmark: the same objects and value, differing only in polarity ( positive , negated ). Examples omit the shared prefix: “a photo of” for attributes, and “a photo of a ⟨ subject ⟩ and a ⟨ target ⟩ , where” for relations.
Figure 2: NegT2IBench scores a negated depth relation. RF-DETR-XL detects the objects, SAM 2 refines their masks, and Depth-Anything V2 estimates depth. Both objects are present, so the image cannot pass by omission, but the horse is behind the computer mouse and the negated statement fails. Checking each requirement separately lets the verdict name which one failed.
Cost
Cohen’s κ
Per element
Kendall τ
Evaluator
Params ↓
GiB ↓
s/img ↓
whole ↑
pos ↑
neg ↑
Δ
Prec. ↑
Rec. ↑
whole ↑
pos ↑
neg ↑
Δ
InternVL3.5-38B
38B
73.0
3.29
48.4
69.5
52.6
-16.9
81.0
78.6
49.2
59.7
52.8
-6.9
Qwen3-VL-32B
32B
62.8
0.99
48.2
74.3
66.5
-7.8
84.1
82.0
48.6
63.3
58.8
-4.5
VQAScore †
11B
24.2
3.40
33.6
49.9
48.6
-1.3
68.4
65.0
38.8
42.3
42.8
0.5
Qwen3-VL-8B
8B
16.8
0.49
48.2
69.7
54.1
-15.6
81.7
78.2
46.4
61.0
54.8
-6.2
InternVL3-8B
8B
15.8
0.71
25.2
59.7
41.8
-17.9
74.7
71.8
46.2
56.3
51.1
-5.2
Table 2: Cost and agreement with human labels on 600 images, against the majority of three annotators. Judge modes: whole prompt (whole), per-element questions (pos), and per-element with negated statements asked in negated form (neg); Δ=neg−pos . Precision and recall use the per-element mode. H. Inter-annotator: agreement between annotators. Bold: best in column. † Timed through t2v-metrics, which scores every image against every question, so time is overestimated.
Statement polarity and load
Semantic category
Load 1
Load 2
Load 3
Load 4
Attributes
Relations
Model
1:0
0:1
2:0
1:1
0:2
2:1
1:2
2:2
Color
Material
Compar.
Depth
Direct.
Proxim.
Overall
U-Net diffusion
SD 1.5
36.1
44.1
7.3
9.7
14.0
1.8
2.4
0.5
27.7
22.8
8.9
8.8
9.6
9.1
14.5
SD 1.5 + NegationCLIP
35.7
44.3
8.8
9.6
15.5
1.8
3.0
0.5
28.3
23.9
8.4
9.4
10.0
9.2
14.9
Diffusion / flow transformers
Table 3: Accuracy (%) on NegT2IBench , by polarity and statement load (left) and by semantic category (right). Even the strongest models struggle with negation, and accuracy falls sharply as load increases. P:N denotes P asserted and N negated statements; category values are averaged over all eight settings. Bold marks the best value per column.
Figure 3: Statement-level failure decomposition, pooled over all models. Each bar partitions the failed statements of one polarity setting (a) or category (b) into the five failure causes; the number above each bar is its count of failed statements. Bars show how failures divide among causes, not how often statements fail. Each failed statement is counted once; statements that were satisfied are not counted, even in failed images. Dead zones only affect positive statements in directional, comparative, and depth relations, so the other bars have no dead-zone failures.
Figure 4: Qualitative comparison of positive and negated prompts. Each column shows one model and a different semantic category: proximity, material, comparative, directional, color, and depth. The top row contains single positive statements ( 1:0 ); the bottom row negates the corresponding statements ( 0:1 ), keeping the objects and tested predicates unchanged. Marks give the NegT2IBench verdict (✓ pass, ✗ fail). In all six pairs, the model satisfies the positive prompt and fails the negated one.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Setting
Prompt
Color
pos0_neg1
a photo of a non-brown orange
Material
pos2_neg2
a photo of a plastic bus, a plastic knife, a non-plastic sandwich, and a non-plastic toilet
Comparative
pos0_neg1
a photo of a skis and a skateboard, where the skis is not bigger than the skateboard
Depth
pos0_neg1
a photo of a broccoli and a bench, where the broccoli is not in front of the bench
Directional
pos2_neg2
a photo of a laptop, a zebra, a backpack, a surfboard, and a carrot, where the laptop is right of the zebra, is right of the backpack, is not below the surfboard, and is not right of the carrot
Proximity
pos0_neg1
a photo of a broccoli and a bench, where the broccoli is not far from the bench
Appendix
Table 4: Representative prompts from the released corpus.
Figure 5: Object distribution per category. Each bar is one object’s occurrence count, sorted from most to least used; the dashed line shows the most even allocation reachable for the observed number of draws. All 79 eligible COCO classes appear in every category.
Figure 6: Predicate and attribute-value balance. Frequency of each relation predicate or attribute value within its category.
Figure 7: The annotation interface. The prompt is shown in full, then one question per required object and one per statement. The prompt here negates two relations, but both are asked affirmatively: the annotator answers whether the backpack is left of the fork, and a correct image gives “no”. Annotators see neither the expected answers nor the T2I generator.
Statements
n
Fleiss’ κ
Unanimous
Leave-one-out κ
All statements
2772
72.4
84.9
70.9
Affirmative
2249
73.0
88.3
73.3
Negated
523
60.2
70.2
59.7
Appendix
Table 5: Agreement between annotators. Fleiss’ κ and the share of statements all three answered alike. Negated statements are harder for people as well, so we compare evaluators against the leave-one-out inter-annotator score rather than against perfect agreement.
Evaluator
Whole prompt
Per element
Negated per element
Detector-based
NegT2IBench (ours)
–
72.0
72.0
VQA judges
VQAScore
33.6
49.9
48.6
mPLUG
7.9
16.0
10.1
Qwen2.5-VL-7B
35.2
52.8
44.3
Appendix
Table 6: Binary agreement, all scoring modes. Whole prompt, the benchmark’s per-element decomposition, and that decomposition with the statements the prompt negates asked in negated form; all other questions are unchanged.
Evaluator
ρ
τ
Pair acc.
Qwen3-VL-32B [ 3 ]
whole prompt
63.2 [57.6, 68.2]
48.6 [43.9, 52.9]
87.5 [81.7, 92.4]
pos. per element
76.5 [71.5, 81.1]
63.3 [58.7, 67.7]
94.7 [91.6, 97.4]
neg. per element
72.4 [67.0, 77.3]
58.8 [54.2, 63.3]
94.4 [91.3, 97.1]
Qwen3-VL-8B [ 3 ]
whole prompt
60.4 [54.2, 66.0]
46.4 [41.2, 51.1]
86.5 [81.4, 91.2]
Appendix
Table 7: Rank agreement with the human score. Spearman ρ , Kendall τ and pairwise ranking accuracy, with 95% bootstrap intervals over images. Judges are ordered by their best ρ . Rank agreement needs no threshold, so it is the only comparison that admits the similarity and reward scores.
Evaluator
Params
s/Q
s/img
Load (s)
GPU GiB
InternVL3.5-38B
38B
0.706
3.29
28.6
73.0
Qwen3-VL-32B
32B
0.212
0.99
21.8
62.8
VQAScore †
11B
0.729
3.40
41.6
24.2
Qwen3-VL-8B
8B
0.105
0.49
12.8
16.8
InternVL3-8B
8B
0.152
0.71
26.3
15.8
InternVL3.5-8B
8B
0.164
0.76
13.8
17.1
Appendix
Table 8: Cost of one evaluation. The same 600 images on one H200 NVL, one process, no batching, eight CPU threads. Model load is reported separately and excluded from s/img. GPU memory is torch.cuda.max_memory_allocated . For a judge, s/img is s/Q multiplied by the 4.66 element questions an image carries on average: every judge is given the benchmark’s own decomposition, so they answer identical questions and differ only in s/Q. NegT2IBench asks no questions and is timed per image. † Scored through t2v-metrics, which computes a K×K image–text matrix and keeps only its diagonal, so the wall-clock reflects that wrapper rather than the model.
Evaluator
TP
TN
FP
FN
Prec.
Rec.
Called correct (%)
Qwen3-VL-32B, pos.
169
362
32
37
84.1
82.0
33.5
Qwen3-VL-8B, pos.
161
358
36
45
81.7
78.2
32.8
InternVL3.5-38B, pos.
162
356
38
44
81.0
78.6
33.3
InternVL3.5-8B, pos.
169
346
48
37
77.9
82.0
36.2
InternVL3-8B, pos.
148
344
50
58
74.7
71.8
33.0
Qwen2.5-VL-7B, pos.
117
364
30
89
79.6
56.8
24.5
Appendix
Table 9: Error structure. Counts against the majority label of three annotators on 600 images, for the per-element scoring mode. FP is an image the annotators reject but the evaluator passes; FN is an image the annotators accept but the evaluator rejects. Judges are ordered by Cohen’s κ ; NegT2IBench is listed separately.
Evaluator
Acc. whole
Acc. pos.
Acc. neg.
Flip (%)
Compl. err.
Called corr. pos. (%)
Called corr. neg. (%)
VQAScore
64.8
77.7
76.7
80.0
0.160
32.7
35.3
Qwen3-VL-32B
73.5
88.5
84.7
77.5
0.225
33.5
36.3
Qwen3-VL-8B
74.2
86.5
77.8
62.1
0.364
32.8
44.5
InternVL3.5-8B
64.7
85.8
76.5
58.9
0.361
36.2
41.2
InternVL3.5-38B
73.2
86.3
77.7
51.1
0.381
33.3
40.7
InternVL3-8B
56.8
82.0
71.5
46.7
0.431
33.0
46.8
Appendix
Table 10: Negated-question probe. Acc. whole, pos., neg.: image accuracy against the binary human label when the judge is asked the whole prompt at once, the per-element questions in positive form, or the same questions with the negated statements asked in negated form. Flip: share of negated statements whose yes/no answer changes between the positive and negated wording. Compl. err.: mean ∣ppos+pneg−1∣ over those statements, where p is the probability of “yes” under each wording; it is zero when the two answers are complementary. Called corr. pos., neg.: share of the 600 images the evaluator passes in each mode; the annotators pass 34.3%. NegT2IBench has one scoring mode and no question to rephrase.
Load 1
Load 2
Load 3
Load 4
Evaluator
1:0
0:1
2:0
1:1
0:2
2:1
1:2
2:2
All
Qwen3-VL-32B [ 3 ]
whole prompt
46.4
57.0
43.8
50.3
38.8
42.9
49.0
30.3
48.6
pos. per element
66.8
66.8
62.3
65.4
67.4
60.2
51.7
57.7
63.3
neg. per element
–
68.5
–
55.5
53.8
57.9
51.5
49.6
58.8
Qwen3-VL-8B [ 3 ]
Appendix
Table 11: Kendall τ against the continuous human score, by polarity setting (positive:negated statements). Judges are shown in their three scoring modes. The inter-annotator score (H. inter-annotator) is recomputed within each setting. Settings with no negated statement (1:0, 2:0) have nothing to rephrase, so their negated row is omitted. Each cell holds 70 to 90 images; per-cell 95% bootstrap intervals have a median width of 32 points. All is computed on all 600 images.
Attributes
Relations
Evaluator
Color
Material
Comparative
Depth
Directional
Proximity
All
Qwen3-VL-32B [ 3 ]
whole prompt
69.7
58.9
47.2
39.9
47.9
38.2
48.6
pos. per element
83.2
69.7
57.3
63.7
61.7
53.2
63.3
neg. per element
83.5
67.8
41.9
59.7
53.2
50.1
58.8
Qwen3-VL-8B [ 3 ]
Appendix
Table 12: Kendall τ against the continuous human score, by semantic category. Judges are shown in their three scoring modes. The inter-annotator score (H. inter-annotator) is recomputed within each category. Each cell holds 100 images; per-cell 95% bootstrap intervals have a median width of 26 points. All is computed on all 600 images.
Evaluator
n
Acc.
κ
Satisfied
Unsatisfied
Balanced
Affirmative statements
NegT2IBench (ours)
2249
90.5
64.3 [59.6, 68.9]
95.6
65.5
80.5
VQAScore, pos.
2242
88.8
53.4 [48.1, 58.4]
96.9
48.9
72.9
mPLUG, pos.
2242
42.5
-1.0 [-3.9, 2.0]
39.2
58.8
49.0
Qwen2.5-VL-7B, pos.
2242
79.0
48.3 [44.4, 52.1]
75.9
94.4
85.1
InternVL3-8B, pos.
2242
90.9
63.8 [59.2, 68.6]
97.2
60.1
78.6
Appendix
Table 13: Element-level agreement, by polarity. One decision per object and per statement instead of one verdict per image. Negated statements are harder for the annotators too, so each block has its own inter-annotator score. The last three columns give the share of satisfied and of unsatisfied statements each method gets right, and their mean. Intervals on κ are 95% bootstrap intervals resampling images.
Category (#values)
What we check
Condition for the attribute or relation to hold
Color (10) / Material (4)
The object carries the requested color / material
sv≥smax−β,(β=0.5 for color, 0 for material)sv is SigLIP 2 score for color / material v on object crop and smax the highest; for color, objects count as having every value within 0.5 of smax , so when two values score closely, a statement denying either fails; material uses the top value only
Directional (4)
The subject lies on the requested side of the target
left / right: dx<−0.45 , dx>0.45 above / below: dy<−0.45 , dy>0.45(dx,dy) is offset between mask centers, shrunk by 0.075× summed mask extents and normalized by original center distance
Proximity (2)
Two objects are near each other or far apart
near: g<0.75R ; far from: g≥0.75Rg is minimum mask-to-mask gap and R is mean equivalent radius. Every measurable pair is either near or far, so proximity has no designed ambiguous middle
Comparative (4)
One object is bigger or taller than another
bigger / smaller: aA>1.05aB ; aB>1.05aA taller / shorter: hA>1.05hB ; hB>1.05hAa is mask area and h mask height; pairs within the factor are called neither way
Depth (2)
The subject is in front of the target, or behind it
Let δˉA and δˉB be the mean disparities over the two objects’ facing surfaces, defined within the mask-to-mask gap plus 16 pixels. A depth relation is emitted only if δmax−δmin∣δˉA−δˉB∣≥0.05+0.10Rg. If the condition holds, the object with larger disparity is in front; otherwise no depth predicate is emitted
Appendix
Table 14: NegT2IBench evaluation rules. Boolean tests on masks by category: positive statements pass when condition holds, negated ones when it does not. Categories span diverse negation forms.
Stage
Setting
Current value
Detection
Backbone
RF-DETR-XL ( RFDETRSegXLarge )
Detection
Presence threshold
0.03
Detection
Counting threshold
0.5
Detection
Per-class cap
16
Detection
Overlap suppression
disabled ( IoU=1.0 )
Masks
Refiner
SAM 2.1 Hiera Small
Appendix
Table 15: Current evaluator configuration used for generator benchmark scoring. Attribute buffers are measured in 100× cosine-similarity units.
Stage
Setting
Previous value
Directional
Extent margin τdir
0.10
Directional
Axis threshold αdir
0.50
Proximity
Gap factor γprox
0.50
Comparative
Size ratio ρsize
1.15
Depth
Rule
contact-band rule
Depth
Contact-band width
12 px
Appendix
Table 16: Previous relation-rule configuration used in the evaluator ablation. These settings replace the corresponding current relation rules; attribute scoring is unchanged.
Config.
Detector
SAM
Constants
n
Agreement (%)
κ
C6
RF-DETR-XL
✓
Current
140
88.5
0.732
C8
RF-DETR-XL
–
Current
140
87.1
0.698
C2
Mask R-CNN
✓
Current
140
85.6
0.654
C4
Mask R-CNN
–
Current
140
84.9
0.640
C7
RF-DETR-XL
–
Previous
140
82.0
0.578
C1
Mask R-CNN
✓
Previous
140
81.3
0.557
Appendix
Table 17: Full evaluator ablation against human annotations.
Model
Steps
Resolution
FLUX.2-klein-9B
4
10242
HunyuanImage-2.1 (distilled)
8
10242
Z-Image-Turbo
9
10242
FLUX.2-dev (4-bit)
28
10242
SD 3.5 Large
28
10242
BLIP3-o NEXT SFT 3B
30
10242
Appendix
Table 18: Generation settings used in the benchmark sweep. Resolutions are pipeline defaults. Janus-Pro-7B is autoregressive, so a diffusion step count is not applicable.
Category
Evaluable
Passed
Dead-zone passes
Share of passes
Depth
30,335
17,947
9,963
55.5%
Directional
29,777
17,117
1,567
9.2%
Comparative
29,883
12,962
374
2.9%
Proximity
30,620
14,764
3
0.0%
Total
120,615
62,790
11,907
19.0%
Appendix
Table 19: Dead-zone exposure among successful negated relations. A dead-zone pass occurs when a negated statement succeeds while the relation evaluator returns no predicate for that object pair.
Figure 8: Qualitative examples. Six selected prompts (one per category) generated by all eleven T2I models.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Text-to-image (T2I) models often produce stereotypical images when stereotype-relevant attributes are left unspecified. We ask whether explicit requests can overcome these defaults, especially when the requested attribute contradicts the stereotype. We introduce \mbox{IMPLICIT-Bench}, a challenge benchmark of 1,831 prompt triplets built on a bias knowledge graph (KG) of stereotypes recorded in StereoSet and CrowS-Pairs. Each triplet describes one scene with a neutral prompt, a stereotype prompt, and an anti-stereotype prompt. The neutral prompt measures the model's default, and the two explicit prompts measure whether the model can follow a request on either side of the stereotype. On this benchmark, Qwen-Image depicts the stereotype in 52% of neutral images and the anti-stereotype in 5%. On both Qwen-Image and Stable Diffusion~3, explicit requests that contradict this default succeed far less often than requests that agree with it. Where Qwen-Image's default is strongest, explicitly requesting the stereotype succeeds in 97% of cases, whereas requesting the anti-stereotype succeeds in only 54%. In these scenes, 31% of the images for anti-stereotype requests show the stereotypical attribute instead. We further test whether prompt rewriting and embedding steering can reduce the default on neutral prompts. With the prompt's own KG unit, the best configurations improve a combined bias and alignment score by up to 9.5 points, and with a unit retrieved from the bias KG, by at most 2.9. Debiasing thus depends on knowing which stereotype a prompt evokes.
Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.
Ibrahim Mohamed Serouis, David Jaramillo Duque
Talan Research and Innovation Center Toulouse, France · Talan Research and Innovation Center Paris, France
Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7% for components with two capabilities to 37.2% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE's 89.51-point estimate with 21.3% less GPU-s/MP.
Shaoan Zhao, Fang Zhao, Xueqiang Guo +9
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · China Unicom Group Co.,Ltd