NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

Omar Elfatairy; Maria A. Bravo; Jessica Bader; Zeynep Akata

cs.CVOct 2, 2026

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

Authors: Omar Elfatairy, Maria A. Bravo, Jessica Bader, Zeynep Akata

Organizations: Technical University of Munich, Helmholtz Munich, Munich Center for Machine Learning

Abstract

Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complementary ability to satisfy negated constraints, for example, generating "a non-red cup." Measuring negation raises challenges not faced by affirmation-based benchmarks and requires careful prompt and evaluation design. We introduce NegT2IBench, a benchmark of 4,800 prompts covering two attribute types and four relation categories. Prompts are organized by polarity: the number of positive statements that must hold and negated statements that must not, each ranging from 0 to 2. Varying the two independently separates the effect of negation from the effect of prompt complexity. Our detector-based scoring is reproducible, auditable, and pinpoints which requirement failed. On 600 images with three-annotator labels, it agrees with humans as closely as vision-language judges up to 30x larger, while using only a fraction of their GPU memory. Across eleven T2I models and 211,200 images, nine score lower on a single negated statement than on a single positive one. Per-statement scoring reveals that the loss is largest for color and near zero for proximity, and that 41.5% of failed statements render exactly what the prompt forbids. Rendering what a prompt asks for and withholding what it forbids are distinct capabilities that an aggregate compositional score cannot distinguish. NegT2IBench measures the latter directly, providing a controlled testbed for diagnosing negation failures and developing methods to overcome them.

Figures & tables

Figure 1: NegT2IBench prompts mix positive (blue) and negated (pink) statements, spanning six categories (above image, right) and organized by the number of each statement (above image, left). Images come from the benchmarked T2I models. The box under each image gives the human verdict; marks show whether each evaluator agrees (✓) or disagrees (✗) with it.

Attributes
Category	Values	Positive example	Negated example
Color (10)	red, orange, yellow, green, blue, purple, pink, brown, black, white	a red cup	a non-red cup
Material (4)	wooden, metal, plastic, glass	a glass cup	a non-glass cup
Relations
Directional (4)	left of, right of, above, below	the backpack is above the parking meter	the backpack is not above the parking meter
Proximity (2)	near, far from	the backpack is far from the parking meter	the backpack is not far from the parking meter

Table 1: NegT2IBench categories. Each category has 800 prompts (100 prompts per polarity setting, 8 settings, 4,800 prompts over 79 object classes). Each row shows a matched pair from the benchmark: the same objects and value, differing only in polarity ( positive , negated ). Examples omit the shared prefix: “a photo of” for attributes, and “a photo of a

\langle

subject

\rangle

and a

\langle

target

\rangle

, where” for relations.

Figure 2: NegT2IBench scores a negated depth relation. RF-DETR-XL detects the objects, SAM 2 refines their masks, and Depth-Anything V2 estimates depth. Both objects are present, so the image cannot pass by omission, but the horse is behind the computer mouse and the negated statement fails. Checking each requirement separately lets the verdict name which one failed.

Evaluator	Params $\downarrow$	GiB $\downarrow$	s/img $\downarrow$	whole $\uparrow$	pos $\uparrow$	neg $\uparrow$	$\Delta$	Prec. $\uparrow$	Rec. $\uparrow$	whole $\uparrow$	pos $\uparrow$	neg $\uparrow$	$\Delta$
	Cost			Cohen’s $\kappa$				Per element		Kendall $\tau$
InternVL3.5-38B	38B	73.0	3.29	48.4	69.5	52.6	-16.9	81.0	78.6	49.2	59.7	52.8	-6.9
Qwen3-VL-32B	32B	62.8	0.99	48.2	74.3	66.5	-7.8	84.1	82.0	48.6	63.3	58.8	-4.5
VQAScore †	11B	24.2	3.40	33.6	49.9	48.6	-1.3	68.4	65.0	38.8	42.3	42.8	0.5
Qwen3-VL-8B	8B	16.8	0.49	48.2	69.7	54.1	-15.6	81.7	78.2	46.4	61.0	54.8	-6.2
InternVL3-8B	8B	15.8	0.71	25.2	59.7	41.8	-17.9	74.7	71.8	46.2	56.3	51.1	-5.2

Table 2: Cost and agreement with human labels on 600 images, against the majority of three annotators. Judge modes: whole prompt (whole), per-element questions (pos), and per-element with negated statements asked in negated form (neg);

\Delta=\text{neg}-\text{pos}

. Precision and recall use the per-element mode. H. Inter-annotator: agreement between annotators. Bold: best in column. † Timed through t2v-metrics, which scores every image against every question, so time is overestimated.

	Statement polarity and load								Semantic category
	Load 1		Load 2			Load 3		Load 4	Attributes		Relations
Model	1:0	0:1	2:0	1:1	0:2	2:1	1:2	2:2	Color	Material	Compar.	Depth	Direct.	Proxim.	Overall
U-Net diffusion
SD 1.5	36.1	44.1	7.3	9.7	14.0	1.8	2.4	0.5	27.7	22.8	8.9	8.8	9.6	9.1	14.5
SD 1.5 + NegationCLIP	35.7	44.3	8.8	9.6	15.5	1.8	3.0	0.5	28.3	23.9	8.4	9.4	10.0	9.2	14.9
Diffusion / flow transformers

Table 3: Accuracy (%) on NegT2IBench , by polarity and statement load (left) and by semantic category (right). Even the strongest models struggle with negation, and accuracy falls sharply as load increases.

P{:}N

denotes

P

asserted and

N

negated statements; category values are averaged over all eight settings. Bold marks the best value per column.

Panel of Figure 3 — Figure 3: Statement-level failure decomposition, pooled over all models. Each bar partitions the failed statements of one polarity setting (a) or category (b) into the five failure causes; the number above each bar is its count of failed statements. Bars show how failures divide among causes, not how often statements fail. Each failed statement is counted once; statements that were satisfied are not counted, even in failed images. Dead zones only affect positive statements in directional, comparative, and depth relations, so the other bars have no dead-zone failures.

Figure 4: Qualitative comparison of positive and negated prompts. Each column shows one model and a different semantic category: proximity, material, comparative, directional, color, and depth. The top row contains single positive statements ( $1{:}0$ ); the bottom row negates the corresponding statements ( $0{:}1$ ), keeping the objects and tested predicates unchanged. Marks give the NegT2IBench verdict (✓ pass, ✗ fail). In all six pairs, the model satisfies the positive prompt and fails the negated one.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Setting	Prompt
Color	pos0_neg1	a photo of a non-brown orange
Material	pos2_neg2	a photo of a plastic bus, a plastic knife, a non-plastic sandwich, and a non-plastic toilet
Comparative	pos0_neg1	a photo of a skis and a skateboard, where the skis is not bigger than the skateboard
Depth	pos0_neg1	a photo of a broccoli and a bench, where the broccoli is not in front of the bench
Directional	pos2_neg2	a photo of a laptop, a zebra, a backpack, a surfboard, and a carrot, where the laptop is right of the zebra, is right of the backpack, is not below the surfboard, and is not right of the carrot
Proximity	pos0_neg1	a photo of a broccoli and a bench, where the broccoli is not far from the bench

Appendix

Table 4: Representative prompts from the released corpus.

Figure 5: Object distribution per category. Each bar is one object’s occurrence count, sorted from most to least used; the dashed line shows the most even allocation reachable for the observed number of draws. All 79 eligible COCO classes appear in every category.

Figure 6: Predicate and attribute-value balance. Frequency of each relation predicate or attribute value within its category.

Figure 7: The annotation interface. The prompt is shown in full, then one question per required object and one per statement. The prompt here negates two relations, but both are asked affirmatively: the annotator answers whether the backpack is left of the fork, and a correct image gives “no”. Annotators see neither the expected answers nor the T2I generator.

Statements	$n$	Fleiss’ $\kappa$	Unanimous	Leave-one-out $\kappa$
All statements	2772	72.4	84.9	70.9
Affirmative	2249	73.0	88.3	73.3
Negated	523	60.2	70.2	59.7

Appendix

Table 5: Agreement between annotators. Fleiss’

\kappa

and the share of statements all three answered alike. Negated statements are harder for people as well, so we compare evaluators against the leave-one-out inter-annotator score rather than against perfect agreement.

Evaluator	Whole prompt	Per element	Negated per element
Detector-based
NegT2IBench (ours)	–	72.0	72.0
VQA judges
VQAScore	33.6	49.9	48.6
mPLUG	7.9	16.0	10.1
Qwen2.5-VL-7B	35.2	52.8	44.3

Appendix

Table 6: Binary agreement, all scoring modes. Whole prompt, the benchmark’s per-element decomposition, and that decomposition with the statements the prompt negates asked in negated form; all other questions are unchanged.

Qwen3-VL-32B [ 3 ]
Evaluator	$\rho$	$\tau$	Pair acc.
whole prompt	63.2 [57.6, 68.2]	48.6 [43.9, 52.9]	87.5 [81.7, 92.4]
pos. per element	76.5 [71.5, 81.1]	63.3 [58.7, 67.7]	94.7 [91.6, 97.4]
neg. per element	72.4 [67.0, 77.3]	58.8 [54.2, 63.3]	94.4 [91.3, 97.1]
Qwen3-VL-8B [ 3 ]
whole prompt	60.4 [54.2, 66.0]	46.4 [41.2, 51.1]	86.5 [81.4, 91.2]

Appendix

Table 7: Rank agreement with the human score. Spearman

\rho

, Kendall

\tau

and pairwise ranking accuracy, with 95% bootstrap intervals over images. Judges are ordered by their best

\rho

. Rank agreement needs no threshold, so it is the only comparison that admits the similarity and reward scores.

Evaluator	Params	s/Q	s/img	Load (s)	GPU GiB
InternVL3.5-38B	38B	0.706	3.29	28.6	73.0
Qwen3-VL-32B	32B	0.212	0.99	21.8	62.8
VQAScore †	11B	0.729	3.40	41.6	24.2
Qwen3-VL-8B	8B	0.105	0.49	12.8	16.8
InternVL3-8B	8B	0.152	0.71	26.3	15.8
InternVL3.5-8B	8B	0.164	0.76	13.8	17.1

Appendix

Table 8: Cost of one evaluation. The same 600 images on one H200 NVL, one process, no batching, eight CPU threads. Model load is reported separately and excluded from s/img. GPU memory is torch.cuda.max_memory_allocated . For a judge, s/img is s/Q multiplied by the

4.66

element questions an image carries on average: every judge is given the benchmark’s own decomposition, so they answer identical questions and differ only in s/Q. NegT2IBench asks no questions and is timed per image. † Scored through t2v-metrics, which computes a

K\times K

image–text matrix and keeps only its diagonal, so the wall-clock reflects that wrapper rather than the model.

Evaluator	TP	TN	FP	FN	Prec.	Rec.	Called correct (%)
Qwen3-VL-32B, pos.	169	362	32	37	84.1	82.0	33.5
Qwen3-VL-8B, pos.	161	358	36	45	81.7	78.2	32.8
InternVL3.5-38B, pos.	162	356	38	44	81.0	78.6	33.3
InternVL3.5-8B, pos.	169	346	48	37	77.9	82.0	36.2
InternVL3-8B, pos.	148	344	50	58	74.7	71.8	33.0
Qwen2.5-VL-7B, pos.	117	364	30	89	79.6	56.8	24.5

Appendix

Table 9: Error structure. Counts against the majority label of three annotators on 600 images, for the per-element scoring mode. FP is an image the annotators reject but the evaluator passes; FN is an image the annotators accept but the evaluator rejects. Judges are ordered by Cohen’s

\kappa

; NegT2IBench is listed separately.

Evaluator	Acc. whole	Acc. pos.	Acc. neg.	Flip (%)	Compl. err.	Called corr. pos. (%)	Called corr. neg. (%)
VQAScore	64.8	77.7	76.7	80.0	0.160	32.7	35.3
Qwen3-VL-32B	73.5	88.5	84.7	77.5	0.225	33.5	36.3
Qwen3-VL-8B	74.2	86.5	77.8	62.1	0.364	32.8	44.5
InternVL3.5-8B	64.7	85.8	76.5	58.9	0.361	36.2	41.2
InternVL3.5-38B	73.2	86.3	77.7	51.1	0.381	33.3	40.7
InternVL3-8B	56.8	82.0	71.5	46.7	0.431	33.0	46.8

Appendix

Table 10: Negated-question probe. Acc. whole, pos., neg.: image accuracy against the binary human label when the judge is asked the whole prompt at once, the per-element questions in positive form, or the same questions with the negated statements asked in negated form. Flip: share of negated statements whose yes/no answer changes between the positive and negated wording. Compl. err.: mean

|p_{\text{pos}}+p_{\text{neg}}-1|

over those statements, where

p

is the probability of “yes” under each wording; it is zero when the two answers are complementary. Called corr. pos., neg.: share of the 600 images the evaluator passes in each mode; the annotators pass 34.3%. NegT2IBench has one scoring mode and no question to rephrase.

Qwen3-VL-32B [ 3 ]
	Load 1		Load 2			Load 3		Load 4
Evaluator	1:0	0:1	2:0	1:1	0:2	2:1	1:2	2:2	All
whole prompt	46.4	57.0	43.8	50.3	38.8	42.9	49.0	30.3	48.6
pos. per element	66.8	66.8	62.3	65.4	67.4	60.2	51.7	57.7	63.3
neg. per element	–	68.5	–	55.5	53.8	57.9	51.5	49.6	58.8
Qwen3-VL-8B [ 3 ]

Appendix

Table 11: Kendall

\tau

against the continuous human score, by polarity setting (positive:negated statements). Judges are shown in their three scoring modes. The inter-annotator score (H. inter-annotator) is recomputed within each setting. Settings with no negated statement (1:0, 2:0) have nothing to rephrase, so their negated row is omitted. Each cell holds 70 to 90 images; per-cell 95% bootstrap intervals have a median width of 32 points. All is computed on all 600 images.

Qwen3-VL-32B [ 3 ]
	Attributes		Relations
Evaluator	Color	Material	Comparative	Depth	Directional	Proximity	All
whole prompt	69.7	58.9	47.2	39.9	47.9	38.2	48.6
pos. per element	83.2	69.7	57.3	63.7	61.7	53.2	63.3
neg. per element	83.5	67.8	41.9	59.7	53.2	50.1	58.8
Qwen3-VL-8B [ 3 ]

Appendix

Table 12: Kendall

\tau

against the continuous human score, by semantic category. Judges are shown in their three scoring modes. The inter-annotator score (H. inter-annotator) is recomputed within each category. Each cell holds 100 images; per-cell 95% bootstrap intervals have a median width of 26 points. All is computed on all 600 images.

Affirmative statements
Evaluator	$n$	Acc.	$\kappa$	Satisfied	Unsatisfied	Balanced
NegT2IBench (ours)	2249	90.5	64.3 [59.6, 68.9]	95.6	65.5	80.5
VQAScore, pos.	2242	88.8	53.4 [48.1, 58.4]	96.9	48.9	72.9
mPLUG, pos.	2242	42.5	-1.0 [-3.9, 2.0]	39.2	58.8	49.0
Qwen2.5-VL-7B, pos.	2242	79.0	48.3 [44.4, 52.1]	75.9	94.4	85.1
InternVL3-8B, pos.	2242	90.9	63.8 [59.2, 68.6]	97.2	60.1	78.6

Appendix

Table 13: Element-level agreement, by polarity. One decision per object and per statement instead of one verdict per image. Negated statements are harder for the annotators too, so each block has its own inter-annotator score. The last three columns give the share of satisfied and of unsatisfied statements each method gets right, and their mean. Intervals on

\kappa

are 95% bootstrap intervals resampling images.

Category (#values)	What we check	Condition for the attribute or relation to hold
Color (10) / Material (4)	The object carries the requested color / material	$s_{v}\geq s_{\max}-\beta,(\beta=0.5\text{ for color, }0\text{ for material})$ $s_{v}$ is SigLIP 2 score for color / material $v$ on object crop and $s_{\max}$ the highest; for color, objects count as having every value within $0.5$ of $s_{\max}$ , so when two values score closely, a statement denying either fails; material uses the top value only
Directional (4)	The subject lies on the requested side of the target	left / right: $d_{x}<-0.45$ , $d_{x}>0.45$ above / below: $d_{y}<-0.45$ , $d_{y}>0.45$ $(d_{x},d_{y})$ is offset between mask centers, shrunk by $0.075\times$ summed mask extents and normalized by original center distance
Proximity (2)	Two objects are near each other or far apart	near: $g<0.75R$ ; far from: $g\geq 0.75R$ $g$ is minimum mask-to-mask gap and $R$ is mean equivalent radius. Every measurable pair is either near or far, so proximity has no designed ambiguous middle
Comparative (4)	One object is bigger or taller than another	bigger / smaller: $a_{A}>1.05a_{B}$ ; $a_{B}>1.05a_{A}$ taller / shorter: $h_{A}>1.05h_{B}$ ; $h_{B}>1.05h_{A}$ $a$ is mask area and $h$ mask height; pairs within the factor are called neither way
Depth (2)	The subject is in front of the target, or behind it	Let $\bar{\delta}_{A}$ and $\bar{\delta}_{B}$ be the mean disparities over the two objects’ facing surfaces, defined within the mask-to-mask gap plus $16$ pixels. A depth relation is emitted only if $\frac{\|\bar{\delta}_{A}-\bar{\delta}_{B}\|}{\delta_{\max}-\delta_{\min}}\geq 0.05+0.10\frac{g}{R}.$ If the condition holds, the object with larger disparity is in front; otherwise no depth predicate is emitted

Appendix

Table 14: NegT2IBench evaluation rules. Boolean tests on masks by category: positive statements pass when condition holds, negated ones when it does not. Categories span diverse negation forms.

Stage	Setting	Current value
Detection	Backbone	RF-DETR-XL ( RFDETRSegXLarge )
Detection	Presence threshold	0.03
Detection	Counting threshold	0.5
Detection	Per-class cap	16
Detection	Overlap suppression	disabled ( $\mathrm{IoU}=1.0$ )
Masks	Refiner	SAM 2.1 Hiera Small

Appendix

Table 15: Current evaluator configuration used for generator benchmark scoring. Attribute buffers are measured in

100\times

cosine-similarity units.

Stage	Setting	Previous value
Directional	Extent margin $\tau_{\mathrm{dir}}$	0.10
Directional	Axis threshold $\alpha_{\mathrm{dir}}$	0.50
Proximity	Gap factor $\gamma_{\mathrm{prox}}$	0.50
Comparative	Size ratio $\rho_{\mathrm{size}}$	1.15
Depth	Rule	contact-band rule
Depth	Contact-band width	12 px

Appendix

Table 16: Previous relation-rule configuration used in the evaluator ablation. These settings replace the corresponding current relation rules; attribute scoring is unchanged.

Config.	Detector	SAM	Constants	$n$	Agreement (%)	$\kappa$
C6	RF-DETR-XL	✓	Current	140	88.5	0.732
C8	RF-DETR-XL	–	Current	140	87.1	0.698
C2	Mask R-CNN	✓	Current	140	85.6	0.654
C4	Mask R-CNN	–	Current	140	84.9	0.640
C7	RF-DETR-XL	–	Previous	140	82.0	0.578
C1	Mask R-CNN	✓	Previous	140	81.3	0.557

Appendix

Table 17: Full evaluator ablation against human annotations.

Model	Steps	Resolution
FLUX.2-klein-9B	4	$1024^{2}$
HunyuanImage-2.1 (distilled)	8	$1024^{2}$
Z-Image-Turbo	9	$1024^{2}$
FLUX.2-dev (4-bit)	28	$1024^{2}$
SD 3.5 Large	28	$1024^{2}$
BLIP3-o NEXT SFT 3B	30	$1024^{2}$

Appendix

Table 18: Generation settings used in the benchmark sweep. Resolutions are pipeline defaults. Janus-Pro-7B is autoregressive, so a diffusion step count is not applicable.

Category	Evaluable	Passed	Dead-zone passes	Share of passes
Depth	30,335	17,947	9,963	55.5%
Directional	29,777	17,117	1,567	9.2%
Comparative	29,883	12,962	374	2.9%
Proximity	30,620	14,764	3	0.0%
Total	120,615	62,790	11,907	19.0%

Appendix

Table 19: Dead-zone exposure among successful negated relations. A dead-zone pass occurs when a negated statement succeeds while the relation evaluator returns no predicate for that object pair.

Figure 8: Qualitative examples. Six selected prompts (one per category) generated by all eleven T2I models.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 303 papers

IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation
Sep 21, 2026Yue Dai, Ziyang Liu, Marc Cheong +1T2I Generation Demographic Bias in T2I Models
The University of Melbourne
Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation
Aug 29, 2026Ibrahim Mohamed Serouis, David Jaramillo DuqueT2I Generation Evaluation Compositional T2I Generation
Talan Research and Innovation Center Toulouse, France · Talan Research and Innovation Center Paris, France
Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
Aug 25, 2026Shaoan Zhao, Fang Zhao, Xueqiang Guo +9T2I Generation Evaluation Compositional T2I Generation
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · China Unicom Group Co.,Ltd

Sep 21, 2026J/K move · Enter open · S save

IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation

Yue Dai, Ziyang Liu, Marc Cheong, Caren Han

The University of Melbourne

T2I Generation Demographic Bias in T2I Models T2I Generation Evaluation

Open paper AskPDF arXiv

Text-to-image (T2I) models often produce stereotypical images when stereotype-relevant attributes are left unspecified. We ask whether explicit requests can overcome these defaults, especially when the requested attribute contradicts the stereotype. We introduce \mbox{IMPLICIT-Bench}, a challenge benchmark of 1,831 prompt triplets built on a bias knowledge graph (KG) of stereotypes recorded in StereoSet and CrowS-Pairs. Each triplet describes one scene with a neutral prompt, a stereotype prompt, and an anti-stereotype prompt. The neutral prompt measures the model's default, and the two explicit prompts measure whether the model can follow a request on either side of the stereotype. On this benchmark, Qwen-Image depicts the stereotype in 52% of neutral images and the anti-stereotype in 5%. On both Qwen-Image and Stable Diffusion~3, explicit requests that contradict this default succeed far less often than requests that agree with it. Where Qwen-Image's default is strongest, explicitly requesting the stereotype succeeds in 97% of cases, whereas requesting the anti-stereotype succeeds in only 54%. In these scenes, 31% of the images for anti-stereotype requests show the stereotypical attribute instead. We further test whether prompt rewriting and embedding steering can reduce the default on neutral prompts. With the prompt's own KG unit, the best configurations improve a combined bias and alignment score by up to 9.5 points, and with a unit retrieved from the bias KG, by at most 2.9. Debiasing thus depends on knowing which stereotype a prompt evokes.

Abstract

T2I Generation

Explore similar work

IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation

Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation

Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing