When Chekhov's Gun Fires Anyway: Over-Realization in Text-to-Image Models
Organizations: The University of Tokyo · National Institute of Informatics · National Taiwan University · Technische Universität Braunschweig ·
Abstract
In text-to-image generation, a target that the prompt mentions often appears in the generated image in a way the prompt does not call for. We call this failure over-realization. Mental-spaces theory points to the situations where it can happen: under a negation, inside what a character believes or imagines, as the vehicle of a figurative comparison, or behind a viewpoint that hides the target. Prior work has studied only negation in depth, and the conventional alignment and preference metrics do not penalize the failure. We build OverReal to measure it: OverReal-Gen, 698 prompts covering the four situations, on which we measure each generator's over-realization rate; and OverReal-Det, 2,841 generated images human-labeled for whether and how the target is over-realized, on which we verify the automatic judge that produces this rate. Our evaluation of eight text-to-image systems reveals that all of them over-realize, and that conventional metrics such as CLIPScore reward the failure rather than penalize it. Prompt expansion only partly mitigates the failure. Inside the generator, the diffusion backbone attends to a mentioned target almost as much as to a requested one. The cue words that should hold the target back receive less attention than ordinary words.
Figures & tables
| Existence-canceling | Mental-state | Figurative | Perspectival | |||||||||||||||||
| CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | |
| DO | .72 | .32 | 22.9 | .30 | 3.0 | .73 | .71 | 21.8 | .32 | 6.4 | .69 | .74 | 22.0 | .30 | 4.8 | .74 | .74 | 22.5 | .30 | 3.6 |
| SO | — | — | — | — | — | .73 | .69 | 21.8 | .30 | 6.7 | .63 | .74 | 21.5 | .29 | 6.0 | .74 | .72 | 22.3 | .29 | 3.5 |
| I | — | — | — | — | — | .74 | .81 | 21.7 | .31 | 8.3 | .69 | .86 | 21.9 | .29 | 6.9 | .76 | .85 | 22.2 | .30 | 4.3 |
| W | .66 | .85 | 22.9 | .30 | 8.8 | .67 | .59 | 21.3 | .29 | 6.6 | .60 | .75 | 21.2 | .27 | 6.8 | .67 | .59 | 21.9 | .28 | 4.5 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | OverReal-Gen | OverReal-Det | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Raw | Expanded | Gold size | Label distribution (%) | Amb. size | Overall agreement ( ) | ||||||
| Size | Avg. tokens | Size | Avg. tokens | DO | SO | I | W | ||||
| Existence-canceling | 200 | 16.9 | 400 | 112.1 | 204 | 32.4 | 0.5 | 0.0 | 67.2 | 23 | 0.73 |
| Mental-state | 75 | 23.5 | 150 | 124.0 | 239 | 53.6 | 18.0 | 11.3 | 17.2 | 112 | 0.51 |
| Figurative | 237 | 16.4 | 474 | 116.4 | 932 | 36.6 | 28.1 | 2.9 | 32.4 | 299 | 0.65 |
| Perspectival | 186 | 23.1 | 372 | 118.7 | 878 | 70.5 | 10.3 | 7.3 | 12.0 | 154 | 0.72 |
| Existence-canceling | Mental-state | Figurative | Perspectival | All | |
|---|---|---|---|---|---|
| Form of the outcome only | 6 | 52 | 125 | 62 | 245 |
| Over-realized against correct | 13 | 43 | 141 | 69 | 266 |
| One label is other | 4 | 17 | 33 | 23 | 77 |
| Decision | Existence-canceling | Mental-state | Figurative | Perspectival | Overall | |
|---|---|---|---|---|---|---|
| Four labels | AI-human | 1.00 | 0.42 | 0.60 | 0.41 | 0.58 |
| humans | 0.83 | 0.47 | 0.67 | 0.75 | 0.70 | |
| Target appears | AI-human | 1.00 | 0.74 | 0.82 | 0.71 | 0.83 |
| humans | 0.88 | 0.84 | 0.83 | 0.87 | 0.86 | |
| Over-realized vs. correct | AI-human | 1.00 | 0.53 | 0.69 | 0.60 | 0.70 |
| humans | 0.88 | 0.49 | 0.74 | 0.79 | 0.75 |
| Existence-canceling | Mental-state | Figurative | Perspectival | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | CS | VQ | PS | HP | LJ | |
| GPT-Image | .67 | .87 | 23.3 | .31 | 9.4 | .75 | .78 | 22.1 | .33 | 8.2 | .69 | .84 | 22.1 | .31 | 7.5 | .74 | .66 | 22.4 | .29 | 5.6 |
| Nano Banana | .64 | .82 | 22.9 | .28 | 8.7 | .71 | .78 | 21.8 | .30 | 7.8 | .63 | .83 | 21.6 | .28 | 7.5 | .71 | .76 | 22.3 | .28 | 4.5 |
| Ideogram | .61 | .82 | 22.7 | .29 | 8.3 | .66 | .62 | 21.2 | .29 | 6.9 | .57 | .73 | 21.1 | .27 | 6.9 | .70 | .70 | 22.4 | .30 | 3.6 |
| FLUX.1-dev | .66 | .59 | 22.9 | .30 | 5.8 | .73 | .66 | 21.7 | .30 | 6.2 | .62 | .70 | 21.4 | .29 | 5.5 | .73 | .69 | 22.6 | .30 | 3.2 |
| Qwen-Image | .73 | .51 | 23.2 | .31 | 5.2 | .75 | .75 | 22.0 | .32 | 6.7 | .66 | .79 | 21.7 | .30 | 6.3 | .77 | .77 | 22.8 | .31 | 3.3 |
| Existence-canceling | Mental-state | Figurative | Perspectival | |||||||||||||||||
| Generator | DO | SO | I | W | DO | SO | I | W | DO | SO | I | W | DO | SO | I | W | ||||
| (a) OverReal-Gen | ||||||||||||||||||||
| GPT-Image | .02 | .00 | .00 | .98 | .02 | .35 | .26 | .28 | .11 | .08 | .22 | .29 | .16 | .33 | .05 | .54 | .15 | .03 | .28 | .05 |
| Nano Banana | .12 | .00 | .00 | .88 | .03 | .37 | .19 | .37 | .07 | .09 | .15 | .26 | .15 | .43 | .09 | .78 | .17 | .02 | .03 | .07 |
| Ideogram | .08 | .00 | .00 | .92 | .02 | .29 | .24 | .03 | .44 | .02 | .07 | .25 | .03 | .65 | .04 | .65 | .18 | .00 | .17 | .02 |
| FLUX.1-dev | .43 | .00 | .00 | .57 | .02 | .44 | .28 | .03 | .24 | .02 | .09 | .30 | .14 | .46 | .02 | .56 | .19 | .01 | .24 | .03 |
| Existence-canceling | Mental-state | Figurative | Perspectival | |
|---|---|---|---|---|
| GPT-Image | 9.5 | 8.8 | 9.2 | 6.3 |
| Nano Banana | 9.1 | 8.8 | 9.0 | 6.2 |
| Ideogram | 9.0 | 7.8 | 8.2 | 5.2 |
| FLUX.1-dev | 6.2 | 7.7 | 7.9 | 4.9 |
| Qwen-Image | 5.0 | 8.3 | 8.7 | 5.3 |
| OmniGen2 | 6.0 | 7.6 | 8.0 | 5.0 |
| Family | Opus mean | GPT mean | Pearson | Mean | |
|---|---|---|---|---|---|
| Existence-canceling | 6.86 2.83 | 7.37 3.27 | 0.90 | 1.02 | 1700 |
| Mental-state | 6.78 1.52 | 7.97 1.30 | 0.62 | 1.32 | 1750 |
| Figurative | 6.71 1.95 | 8.29 1.39 | 0.57 | 1.69 | 1800 |
| Perspectival | 3.63 1.81 | 5.27 2.38 | 0.56 | 1.98 | 1800 |
| Overall | 5.98 2.49 | 7.22 2.52 | 0.78 | 1.51 | 7050 |
| Existence-canceling | Mental-state | Figurative | Perspectival | All families | |
|---|---|---|---|---|---|
| GPT-Image | .02 .03 | .61 .12 | .51 .11 | .69 .10 | .43 .05 |
| Nano Banana | .12 .09 | .56 .11 | .41 .10 | .95 .04 | .49 .05 |
| Ideogram | .08 .07 | .53 .11 | .32 .09 | .83 .08 | .41 .04 |
| FLUX.1-dev | .43 .13 | .73 .11 | .39 .10 | .75 .10 | .53 .06 |
| Qwen-Image | .64 .13 | .89 .07 | .62 .10 | .89 .06 | .73 .05 |
| OmniGen2 | .51 .11 | .86 .08 | .45 .11 | .81 .08 | .61 .05 |
| Existence-canceling | Mental-state | Figurative | Perspectival | All | |
|---|---|---|---|---|---|
| .84 | .77 | .65 | .71 | .79 | |
| .00 | .17 | .10 | .20 | .10 | |
| .16 | .06 | .26 | .09 | .12 | |
| — | .44 | .05 | .64 | .33 | |
| — | .50 | .76 | .32 | .57 | |
| — | .06 | .19 | .04 | .10 |
| by kind of over-realization | by family | ||||||
|---|---|---|---|---|---|---|---|
| Metric | Winning rate | Disruptive | Silent | Existence-canceling | Mental-state | Figurative | Perspectival |
| CLIPScore | .64 .10 | .69 | .44 | .75 | .56 | .60 | .38 |
| VQAScore | .33 .13 | .25 | .80 | .05 | .67 | .75 | .50 |
| PickScore | .66 .10 | .72 | .44 | .72 | .67 | .60 | .50 |
| HPSv2 | .65 .11 | .69 | .50 | .65 | .72 | .70 | .38 |
| LLM judge | .12 .08 | .06 | .50 | .00 | .27 | .33 | .17 |
| OverReal-Gen | In the wild | |
|---|---|---|
| Existence-canceling | 1.00 | 1.00 |
| Mental-state | .69 | .66 |
| Figurative | .35 | .08 |
| Perspectival | .79 | .84 |
| , four labels | , OR vs. correct | OR rate in the wild | |||||
|---|---|---|---|---|---|---|---|
| Family | wild | Det, AI-human | Det, humans | wild | automatic | reweighted | |
| Existence-canceling | 34 | .82 | 1.00 | .83 | .82 | .60 | .51 |
| Mental-state | 34 | .76 | .42 | .47 | .76 | .51 | .42 |
| Figurative | 34 | .53 | .60 | .67 | .53 | .28 | .26 |
| Perspectival | 100 | .65 | .41 | .75 | .69 | .80 | .68 |
| All | 202 | .70 | .58 | .70 | .72 | — | — |
| Family | Rewriter | Dropped | Excluded | Mentioned | Depicted |
|---|---|---|---|---|---|
| Existence-canceling | Qwen | 0 | 98 | 2 | 0 |
| Ideogram | 95 | 2 | 2 | 0 | |
| Mental-state | Qwen | 0 | 1 | 64 | 35 |
| Ideogram | 41 | 0 | 40 | 19 | |
| Figurative | Qwen | 15 | 0 | 77 | 8 |
| Ideogram | 55 | 0 | 36 | 9 |
| Family | Dropped | Excluded | Mentioned | Depicted |
|---|---|---|---|---|
| Existence-canceling | 0 | 35 | 5 | — |
| Mental-state | 4 | — | 71 | 82 |
| Figurative | 4 | — | 33 | 75 |
| Perspectival | 2 | 100 | 86 | 87 |
| Overall | 2 | 38 | 60 | 84 |
| Encoder | Existence-canceling | Mental-state | Figurative | Perspectival | All |
|---|---|---|---|---|---|
| CLIP-L/14 | .95 .04 | .81 .06 | .87 .03 | .53 .02 | .79 .02 |
| CLIP-bigG/14 | .96 .03 | .87 .05 | .87 .03 | .60 .03 | .82 .02 |
| T5-XXL | .95 .04 | .89 .04 | .94 .02 | .62 .03 | .85 .02 |
| Qwen2.5-VL-7B | 1.00 .00 | .92 .04 | .91 .02 | .60 .03 | .86 .01 |
| Family | Original prompt (no) | Plain-mention control (yes) | Both |
|---|---|---|---|
| Existence-canceling | 1.00 | .76 | .88 |
| Mental-state | .75 | 1.00 | .87 |
| Figurative | .84 | .93 | .89 |
| Perspectival | .56 | .85 | .70 |
| Family | Original prompt | Plain-mention control |
|---|---|---|
| Existence-canceling | a cat sitting on a couch with a dog on the floor, with no book in sight | a cat sitting on a couch with a dog on the floor, with a book in sight |
| Mental-state | Tim sits alone at a classroom desk, remembering the toy robot he lost years ago. | Tim sits alone at a classroom desk, holding the toy robot he got years ago. |
| Figurative | The coach is as fierce as a tiger . | The coach is standing next to a tiger . |
| Perspectival | A desk drawer is completely closed. Inside is a green frog toy. | A desk drawer is pulled open. Inside is a green frog toy. |
| Original prompt | Plain-mention control | Ratios | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Family | target | cue | other | target | replacing | other | |||
| FLUX.1-dev | Existence-canceling | 4.0 | 4.0 | 5.6 | 5.1 | 3.8 | 5.4 | .79 | 1.15 | .71 |
| Mental-state | 3.8 | 3.0 | 4.2 | 4.8 | 3.5 | 4.2 | .74 | .87 | .73 | |
| Figurative | 5.6 | 4.7 | 6.1 | 6.5 | 5.6 | 5.7 | .83 | .80 | .77 | |
| Perspectival | 3.6 | 3.7 | 4.4 | 3.7 | 3.6 | 4.5 | .98 | 1.04 | .83 | |
| All families | 4.2 | 3.8 | 5.1 | 5.0 | 4.1 | 4.9 | .83 | .97 | .76 | |
| FLUX.1-dev | SD3.5-Large | Qwen-Image | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Family | Qwen | Ideo. | orig. | Qwen | Ideo. | orig. | Qwen | Ideo. | orig. | Qwen | Ideo. |
| Existence-canceling | 100 | 2 | .77 | 1.05 | — | .85 | .99 | — | .82 | .58 | — |
| Mental-state | 136 | 76 | .96 | 1.60 | 1.85 | 1.13 | 1.05 | 1.42 | .81 | 1.32 | 1.05 |
| Figurative | 398 | 196 | 1.00 | 1.37 | 1.35 | 1.40 | 1.09 | 1.14 | .78 | 1.43 | 1.17 |
| Perspectival | 282 | 232 | .87 | 1.30 | 1.15 | .94 | 1.00 | 1.06 | .74 | 1.14 | 1.01 |
| All families | 916 | 506 | .90 | 1.33 | 1.45 | 1.08 | 1.04 | 1.21 | .79 | 1.12 | 1.08 |
| Mental-state | Figurative | |||||||||||
| Generator + prompt | Target | DO | SO | I | W | Target | DO | SO | I | W | ||
| GPT-Image-1.5 (raw) | 100 | .89 | .35 | .26 | .28 | .11 | 42 | .57 | .14 | .26 | .17 | .43 |
| + constraint | 70 | .14 | .01 | .03 | .10 | .86 | 115 | .08 | .00 | .05 | .03 | .92 |
| FLUX.1-dev (raw) | 99 | .76 | .44 | .28 | .03 | .24 | 47 | .47 | .09 | .23 | .15 | .53 |
| + constraint | 74 | .77 | .50 | .23 | .04 | .23 | 116 | .59 | .15 | .37 | .07 | .41 |
| Qwen-Image (raw) | 99 | .90 | .67 | .22 | .01 | .10 | 45 | .69 | .27 | .38 | .04 | .31 |
T2I Generation
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
See all 329 papers