Organizations: The University of Tokyo · National Institute of Informatics · National Taiwan University · Technische Universität Braunschweig · University of Illinois Urbana-Champaign · Independent researcher · Kyoto University
In text-to-image generation, a target that the prompt mentions often appears in the generated image in a way the prompt does not call for. We call this failure over-realization. Mental-spaces theory points to the situations where it can happen: under a negation, inside what a character believes or imagines, as the vehicle of a figurative comparison, or behind a viewpoint that hides the target. Prior work has studied only negation in depth, and the conventional alignment and preference metrics do not penalize the failure. We build OverReal to measure it: OverReal-Gen, 698 prompts covering the four situations, on which we measure each generator's over-realization rate; and OverReal-Det, 2,841 generated images human-labeled for whether and how the target is over-realized, on which we verify the automatic judge that produces this rate. Our evaluation of eight text-to-image systems reveals that all of them over-realize, and that conventional metrics such as CLIPScore reward the failure rather than penalize it. Prompt expansion only partly mitigates the failure. Inside the generator, the diffusion backbone attends to a mentioned target almost as much as to a requested one. The cue words that should hold the target back receive less attention than ordinary words.
Figures & tables
Figure 1: Two over-realizations, both generated by Gemini, with the prompt under each image. In (a) the model turns the page to face the camera: the words cannot reach the reader, so the scene contradicts itself. In (b) the model puts a tiger on the team’s emblem and banner, which is not what the comparison meant, but the image still holds together.
Figure 2: Four possible outcomes for the prompt “The coach is as fierce as a tiger .” The target word tiger is mentioned only as the vehicle of a figurative comparison, so it is not strictly requested in the scene. (b) is the outcome the model actually produced. (a) and (b) are over-realization; (c) and (d) are correct.
Figure 3: Main results: over-realization rate of each generator by prompt family, on the raw prompts of OverReal-Gen (top) and on the in-the-wild prompts (bottom). Bar height is the over-realization rate; the dark part is disruptive (DO) and the light part silent (SO) over-realization; red bars are the proprietary generators, blue bars the open models. The full outcome shares, including integrated (I) and withheld (W) realization and the unprompted appearance rate u , are in Table 7 (Appendix D ).
Existence-canceling
Mental-state
Figurative
Perspectival
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
DO
.72
.32
22.9
.30
3.0
.73
.71
21.8
.32
6.4
.69
.74
22.0
.30
4.8
.74
.74
22.5
.30
3.6
SO
—
—
—
—
—
.73
.69
21.8
.30
6.7
.63
.74
21.5
.29
6.0
.74
.72
22.3
.29
3.5
I
—
—
—
—
—
.74
.81
21.7
.31
8.3
.69
.86
21.9
.29
6.9
.76
.85
22.2
.30
4.3
W
.66
.85
22.9
.30
8.8
.67
.59
21.3
.29
6.6
.60
.75
21.2
.27
6.8
.67
.59
21.9
.28
4.5
Table 1: Mean score of the conventional metrics over the images with each label, pooled over generators; cells with fewer than 10 images are left blank.
Figure 4: What the rewriters do with the target. (a) Proportion of rewritten prompts in which the target is dropped, explicitly excluded, mentioned, or explicitly to be depicted. (b) Over-realization rate of the images generated from those prompts, by the four categories, pooled over rewriters and open generators. Full numbers in Tables 16 and 17 .
Figure 5: Histograms: the distribution over the attention of the target words (green) and of the cue words (red). Lines: the over-realization rate (OR rate) within each interval of the share; intervals with fewer than 12 images are not drawn.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Table 2: Taxonomy examples.
Figure 6: Borderline cases: (a) disruptive or silent, (b) disruptive or integrated, (c) integrated or silent. The two labels under each image are the two annotators’ labels.
Figure 7: Two pilot images (Gemini) that over-realize two targets each: the vehicle of the comparison, silently, and the viewpoint of the person, disruptively. (a) The student faces the camera instead of the lecturer. (b) The screen faces the camera instead of the woman reading it.
Family
OverReal-Gen
OverReal-Det
Raw
Expanded
Gold size
Label distribution (%)
Amb. size
Overall agreement ( κ )
Size
Avg. tokens
Size
Avg. tokens
DO
SO
I
W
Existence-canceling
200
16.9
400
112.1
204
32.4
0.5
0.0
67.2
23
0.73
Mental-state
75
23.5
150
124.0
239
53.6
18.0
11.3
17.2
112
0.51
Figurative
237
16.4
474
116.4
932
36.6
28.1
2.9
32.4
299
0.65
Perspectival
186
23.1
372
118.7
878
70.5
10.3
7.3
12.0
154
0.72
Appendix
Table 3: OverReal statistics. Label distribution: share of the gold split labeled disruptive over-realization (DO), silent over-realization (SO), integrated realization (I) and withheld realization (W).
Existence-canceling
Mental-state
Figurative
Perspectival
All
Form of the outcome only
6
52
125
62
245
Over-realized against correct
13
43
141
69
266
One label is other
4
17
33
23
77
Appendix
Table 4: The ambiguous split of OverReal-Det by the kind of disagreement.
Figure 8: The annotation cascade as a decision tree. Each decision node (blue) is one yes/no question put to the annotator in a separate call, with the image and the prompt; the leaves (italic) are the labels. The family-specific wording of the questions is given in Appendix E.1 .
Decision
Existence-canceling
Mental-state
Figurative
Perspectival
Overall
Four labels
AI-human
1.00
0.42
0.60
0.41
0.58
humans
0.83
0.47
0.67
0.75
0.70
Target appears
AI-human
1.00
0.74
0.82
0.71
0.83
humans
0.88
0.84
0.83
0.87
0.86
Over-realized vs. correct
AI-human
1.00
0.53
0.69
0.60
0.70
humans
0.88
0.49
0.74
0.79
0.75
Appendix
Table 5: Cohen’s κ of the automatic annotator (Claude Opus 5) against the human gold labels on the held-out test split of OverReal-Det (1,118 images), and of the two human annotators against each other on the 2,688 doubly annotated images with one label each, for the four-outcome label, for whether the target appears, and for the collapse to over-realized against correct.
Existence-canceling
Mental-state
Figurative
Perspectival
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
CS ↑
VQ ↑
PS ↑
HP ↑
LJ ↑
GPT-Image
.67
.87
23.3
.31
9.4
.75
.78
22.1
.33
8.2
.69
.84
22.1
.31
7.5
.74
.66
22.4
.29
5.6
Nano Banana
.64
.82
22.9
.28
8.7
.71
.78
21.8
.30
7.8
.63
.83
21.6
.28
7.5
.71
.76
22.3
.28
4.5
Ideogram
.61
.82
22.7
.29
8.3
.66
.62
21.2
.29
6.9
.57
.73
21.1
.27
6.9
.70
.70
22.4
.30
3.6
FLUX.1-dev
.66
.59
22.9
.30
5.8
.73
.66
21.7
.30
6.2
.62
.70
21.4
.29
5.5
.73
.69
22.6
.30
3.2
Qwen-Image
.73
.51
23.2
.31
5.2
.75
.75
22.0
.32
6.7
.66
.79
21.7
.30
6.3
.77
.77
22.8
.31
3.3
Appendix
Table 6: Conventional evaluation of the eight systems on the raw prompts of OverReal-Gen: mean CLIPScore (CS), VQAScore (VQ), PickScore (PS), HPSv2 (HP) and LLM judge (LJ) per family.
Existence-canceling
Mental-state
Figurative
Perspectival
Generator
DO ↓
SO ↓
I ↑
W ↑
u↓
DO ↓
SO ↓
I ↑
W ↑
u↓
DO ↓
SO ↓
I ↑
W ↑
u↓
DO ↓
SO ↓
I ↑
W ↑
u↓
(a) OverReal-Gen
GPT-Image
.02
.00
.00
.98
.02
.35
.26
.28
.11
.08
.22
.29
.16
.33
.05
.54
.15
.03
.28
.05
Nano Banana
.12
.00
.00
.88
.03
.37
.19
.37
.07
.09
.15
.26
.15
.43
.09
.78
.17
.02
.03
.07
Ideogram
.08
.00
.00
.92
.02
.29
.24
.03
.44
.02
.07
.25
.03
.65
.04
.65
.18
.00
.17
.02
FLUX.1-dev
.43
.00
.00
.57
.02
.44
.28
.03
.24
.02
.09
.30
.14
.46
.02
.56
.19
.01
.24
.03
Appendix
Table 7: Main results: proportion of images under each outcome label (definition in § 3.2 ), namely disruptive over-realization (DO), silent over-realization (SO), integrated realization (I) and withheld realization (W), and the mean unprompted appearance rate ( u , definition in § 5.3 ). (a) On the curated prompts of OverReal-Gen. (b) On the 397 in-the-wild prompts of § 6.1 , one image per prompt and system; u was not measured there.
Existence-canceling
Mental-state
Figurative
Perspectival
GPT-Image
9.5
8.8
9.2
6.3
Nano Banana
9.1
8.8
9.0
6.2
Ideogram
9.0
7.8
8.2
5.2
FLUX.1-dev
6.2
7.7
7.9
4.9
Qwen-Image
5.0
8.3
8.7
5.3
OmniGen2
6.0
7.6
8.0
5.0
Appendix
Table 8: LLM judge (LJ) ratings by GPT-5.2, the same cells as the LJ column of Table 6 .
Family
Opus mean
GPT mean
Pearson r
Mean ∣Δ∣
n
Existence-canceling
6.86 ± 2.83
7.37 ± 3.27
0.90
1.02
1700
Mental-state
6.78 ± 1.52
7.97 ± 1.30
0.62
1.32
1750
Figurative
6.71 ± 1.95
8.29 ± 1.39
0.57
1.69
1800
Perspectival
3.63 ± 1.81
5.27 ± 2.38
0.56
1.98
1800
Overall
5.98 ± 2.49
7.22 ± 2.52
0.78
1.51
7050
Appendix
Table 9: Between-judge statistics: mean rating ± standard deviation per judge, correlation, and mean absolute rating difference.
Existence-canceling
Mental-state
Figurative
Perspectival
All families
GPT-Image
.02 ± .03
.61 ± .12
.51 ± .11
.69 ± .10
.43 ± .05
Nano Banana
.12 ± .09
.56 ± .11
.41 ± .10
.95 ± .04
.49 ± .05
Ideogram
.08 ± .07
.53 ± .11
.32 ± .09
.83 ± .08
.41 ± .04
FLUX.1-dev
.43 ± .13
.73 ± .11
.39 ± .10
.75 ± .10
.53 ± .06
Qwen-Image
.64 ± .13
.89 ± .07
.62 ± .10
.89 ± .06
.73 ± .05
OmniGen2
.51 ± .11
.86 ± .08
.45 ± .11
.81 ± .08
.61 ± .05
Appendix
Table 10: Over-realization rate of each system per family and over all families, weighting the families by their prompt counts in OverReal-Gen, ± half the width of a 95% bootstrap interval over prompts.
Existence-canceling
Mental-state
Figurative
Perspectival
All
P(DO1∣DO0)
.84
.77
.65
.71
.79
P(SO1∣DO0)
.00
.17
.10
.20
.10
P(C1∣DO0)
.16
.06
.26
.09
.12
P(DO1∣SO0)
—
.44
.05
.64
.33
P(SO1∣SO0)
—
.50
.76
.32
.57
P(C1∣SO0)
—
.06
.19
.04
.10
Appendix
Table 11: Label of the other seed’s image given the label of one image of the same prompt and open model, on the raw prompts of OverReal-Gen. P(y1∣y0) is the share of images with label y0 whose other-seed image has label y1 ; DO disruptive, SO silent, C correct (integrated or withheld). Existence-canceling prompts have no silent images.
by kind of over-realization
by family
Metric
Winning rate
Disruptive
Silent
Existence-canceling
Mental-state
Figurative
Perspectival
CLIPScore
.64 ± .10
.69
.44
.75
.56
.60
.38
VQAScore
.33 ± .13
.25
.80
.05
.67
.75
.50
PickScore
.66 ± .10
.72
.44
.72
.67
.60
.50
HPSv2
.65 ± .11
.69
.50
.65
.72
.70
.38
LLM judge
.12 ± .08
.06
.50
.00
.27
.33
.17
Appendix
Table 12: Winning rate: share of matched pairs (same prompt, same open model, two seeds; one image over-realized, one correct) in which the metric scores the over-realized image higher, ± half the width of a 95% bootstrap interval over prompts.
OverReal-Gen
In the wild
Existence-canceling
1.00
1.00
Mental-state
.69
.66
Figurative
.35
.08
Perspectival
.79
.84
Appendix
Table 14: Disruptive share of over-realization, DO / (DO + SO), pooled over the eight systems, on the raw prompts of OverReal-Gen and on the in-the-wild prompts.
κ , four labels
κ , OR vs. correct
OR rate in the wild
Family
n
wild
Det, AI-human
Det, humans
wild
automatic
reweighted
Existence-canceling
34
.82
1.00
.83
.82
.60
.51
Mental-state
34
.76
.42
.47
.76
.51
.42
Figurative
34
.53
.60
.67
.53
.28
.26
Perspectival
100
.65
.41
.75
.69
.80
.68
All
202
.70
.58
.70
.72
—
—
Appendix
Table 15: Human check of the automatic labels in the wild. Cohen’s κ between the automatic annotator and a human annotator on a stratified sample of n wild images, over the four labels and over the collapse to over-realized (OR) against correct, with the AI-human and human-human values on OverReal-Det (Table 5 ) for reference; and the over-realization rate in the wild from the automatic labels and after reweighting them by the human labels.
Family
Rewriter
Dropped
Excluded
Mentioned
Depicted
Existence-canceling
Qwen
0
98
2
0
Ideogram
95
2
2
0
Mental-state
Qwen
0
1
64
35
Ideogram
41
0
40
19
Figurative
Qwen
15
0
77
8
Ideogram
55
0
36
9
Appendix
Table 16: How the rewriters treat the target, as a percentage of the rewritten prompts of each family: the target is dropped, explicitly excluded (“no keyboard is present”), mentioned, or explicitly to be depicted (“visible through a gap”).
Family
Dropped
Excluded
Mentioned
Depicted
Existence-canceling
0
35
5
—
Mental-state
4
—
71
82
Figurative
4
—
33
75
Perspectival
2
100
86
87
Overall
2
38
60
84
Appendix
Table 17: Over-realization rate (percent) of the images generated from rewritten prompts, by how the rewrite treats the target, pooled over both rewriters and the five open generators. Cells with fewer than 10 images are left blank.
Encoder
Existence-canceling
Mental-state
Figurative
Perspectival
All
CLIP-L/14
.95 ± .04
.81 ± .06
.87 ± .03
.53 ± .02
.79 ± .02
CLIP-bigG/14
.96 ± .03
.87 ± .05
.87 ± .03
.60 ± .03
.82 ± .02
T5-XXL
.95 ± .04
.89 ± .04
.94 ± .02
.62 ± .03
.85 ± .02
Qwen2.5-VL-7B
1.00 ± .00
.92 ± .04
.91 ± .02
.60 ± .03
.86 ± .01
Appendix
Table 18: Accuracy of a linear probe that tells an original prompt from its plain-mention control, from the contextual vector of the target word at the layer each pipeline feeds to the backbone, ± half the width of the bootstrap 95% interval. Chance is 50%.
Figure 9: Accuracy of the target-word probe of Table 18 at every layer of each encoder, per family and pooled. Layer 0 is the embedding output; the dashed line marks the layer the pipeline feeds to the backbone. The perspectival family stays near chance at every layer of every encoder.
Family
Original prompt (no)
Plain-mention control (yes)
Both
Existence-canceling
1.00
.76
.88
Mental-state
.75
1.00
.87
Figurative
.84
.93
.89
Perspectival
.56
.85
.70
Appendix
Table 19: Accuracy of Qwen2.5-VL-7B, the text encoder of Qwen-Image, when asked directly whether the target would be visible.
Family
Original prompt
Plain-mention control
Existence-canceling
a cat sitting on a couch with a dog on the floor, with no book in sight
a cat sitting on a couch with a dog on the floor, with a book in sight
Mental-state
Tim sits alone at a classroom desk, remembering the toy robot he lost years ago.
Tim sits alone at a classroom desk, holding the toy robot he got years ago.
Figurative
The coach is as fierce as a tiger .
The coach is standing next to a tiger .
Perspectival
A desk drawer is completely closed. Inside is a green frog toy.
A desk drawer is pulled open. Inside is a green frog toy.
Appendix
Table 20: Plain-mention controls: one example per family. The target word is kept; the words that keep it out of the scene are replaced by words that put it in.
Original prompt
Plain-mention control
Ratios
Model
Family
target at
cue ac
other ao
target at′
replacing ar′
other ao′
ρt=at/at′
ρc=ac/ar′
γ=ac/ao
FLUX.1-dev
Existence-canceling
4.0
4.0
5.6
5.1
3.8
5.4
.79
1.15
.71
Mental-state
3.8
3.0
4.2
4.8
3.5
4.2
.74
.87
.73
Figurative
5.6
4.7
6.1
6.5
5.6
5.7
.83
.80
.77
Perspectival
3.6
3.7
4.4
3.7
3.6
4.5
.98
1.04
.83
All families
4.2
3.8
5.1
5.0
4.1
4.9
.83
.97
.76
Appendix
Table 21: Attention in the backbone, per family and averaged over the four families. a : percentage of the image-to-text attention per token of the target words (t), the cue words (c) and the other words (o) under the original prompt; a′ : the same under the plain-mention control, where the cue words are replaced (r). The ratios pair the original prompt and its control under the same seed.
FLUX.1-dev
SD3.5-Large
Qwen-Image
Family
n Qwen
n Ideo.
orig.
Qwen
Ideo.
orig.
Qwen
Ideo.
orig.
Qwen
Ideo.
Existence-canceling
100
2
.77
1.05
—
.85
.99
—
.82
.58
—
Mental-state
136
76
.96
1.60
1.85
1.13
1.05
1.42
.81
1.32
1.05
Figurative
398
196
1.00
1.37
1.35
1.40
1.09
1.14
.78
1.43
1.17
Perspectival
282
232
.87
1.30
1.15
.94
1.00
1.06
.74
1.14
1.01
All families
916
506
.90
1.33
1.45
1.08
1.04
1.21
.79
1.12
1.08
Appendix
Table 22: Relative attention r of the target words under the original prompt (orig.) and under the expanded prompts of the Qwen rewriter and of Ideogram Magic Prompt (Ideo.), per family and over all families; n is the number of expanded images per family and rewriter. Ideogram Magic Prompt keeps the target in only two existence-canceling prompts, so that cell is left out and the Ideogram mean excludes the family.
Figure 10: Two items of Table 2 under four prompts, SD3.5-Large: the original prompt, its plain-mention control, the Qwen rewrite and the Ideogram Magic Prompt rewrite (top row: the image; bottom row: attention from the image tokens to the target words in the first denoising steps). Top: the cucumber. The Qwen rewrite keeps cucumber only as a color ( “the deep, fresh hue of a ripe cucumber” ); the backbone draws no cucumber and the map stays flat. The Ideogram rewrite drops the word, so there is no map. Bottom: the whiteboard whose “Good luck!” faces away from the camera. Both rewriters keep the words, both images show them, and under all four prompts the attention sits on the writing.
Mental-state
Figurative
Generator + prompt
n
Target
DO
SO
I
W
n
Target
DO
SO
I
W
GPT-Image-1.5 (raw)
100
.89
.35
.26
.28
.11
42
.57
.14
.26
.17
.43
+ constraint
70
.14
.01
.03
.10
.86
115
.08
.00
.05
.03
.92
FLUX.1-dev (raw)
99
.76
.44
.28
.03
.24
47
.47
.09
.23
.15
.53
+ constraint
74
.77
.50
.23
.04
.23
116
.59
.15
.37
.07
.41
Qwen-Image (raw)
99
.90
.67
.22
.01
.10
45
.69
.27
.38
.04
.31
Appendix
Table 23: Outcome distribution on the raw prompts and on the same prompts with the explicit constraint “No { target } should appear in the image.” appended, for the mental-state prompts and a random half of the figurative prompts of OverReal-Gen. Target is the share of images in which the target appears (DO + SO + I); DO disruptive, SO silent, I integrated, W withheld. The raw rows are over the labeled sample of § 5.3 (both seeds), the constraint rows over one image per prompt (seed 0).
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Text-to-image diffusion models often fail to suppress explicitly negated concepts, such as a lamp for "a desk without a lamp." We introduce T2I-NSI, a benchmark and metric for measuring negation sensitivity in text-to-image generation. It pairs each negated prompt with a canonical positive counterpart and factorizes negation by linguistic form (lexical/semantic, morphological, syntactic) and target type (objects, attributes, relations, compositions, and nested cases). The benchmark contains 759,869 paired prompt atoms over a 5,082-object catalog, with 916,272 images (911,982 open-weights, 4,290 closed-source), scored by a Basic CLIP estimator and a Robust ensemble of CLIP, VQA and open-vocabulary detection. Negation sensitivity is low: open-weights generators score NSI <= 0.038, while GPT-Image-1.5 and Gemini-2.5-Flash-Image reach about 0.09. The dominant failure is under-suppression rather than misdirection. Inverse interpretation is essentially absent (0.00% across over 450K atom-model evaluations), whereas models render the concept under both prompts on 8-19% of atoms and only partly suppress it on a further 9-25%; most remaining atoms are too weakly rendered to label. A blinded human study (420 atoms, nine annotators) corroborates the near-absence of inversion and the high persistence rate, and a matched-prompt comparison shows the open-versus-closed gap (~3.5x) is not a prompt-generator artifact. Code and data: https://github.com/IAB-IITJ/T2I-NSI
Akansha Gautam, Aparna Bharati, Mayank Vatsa +1
Indian Institute of Technology Jodhpur, India · Colorado State University, USA
Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.
Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complementary ability to satisfy negated constraints, for example, generating "a non-red cup." Measuring negation raises challenges not faced by affirmation-based benchmarks and requires careful prompt and evaluation design. We introduce NegT2IBench, a benchmark of 4,800 prompts covering two attribute types and four relation categories. Prompts are organized by polarity: the number of positive statements that must hold and negated statements that must not, each ranging from 0 to 2. Varying the two independently separates the effect of negation from the effect of prompt complexity. Our detector-based scoring is reproducible, auditable, and pinpoints which requirement failed. On 600 images with three-annotator labels, it agrees with humans as closely as vision-language judges up to 30x larger, while using only a fraction of their GPU memory. Across eleven T2I models and 211,200 images, nine score lower on a single negated statement than on a single positive one. Per-statement scoring reveals that the loss is largest for color and near zero for proximity, and that 41.5% of failed statements render exactly what the prompt forbids. Rendering what a prompt asks for and withholding what it forbids are distinct capabilities that an aggregate compositional score cannot distinguish. NegT2IBench measures the latter directly, providing a controlled testbed for diagnosing negation failures and developing methods to overcome them.
Omar Elfatairy, Maria A. Bravo, Jessica Bader +1
Technical University of Munich, Helmholtz Munich, Munich Center for Machine Learning