In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.
Figures & tables
Figure 1: Overview of our pipeline. Multiple MLLMs generate captions for the same image, and a text classifier reliably attributes each caption to its source model. These captions are then used to synthesize new images, on which an image classifier performs the same source-identification task. While text-based attribution is highly accurate, image-based attribution fails, which reveals a clear mismatch between caption-space and image-space model signatures.
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Qwen3-VL
Total
99.83
99.78
99.67
–
99.76
99.92
99.90
–
99.73
99.85
99.85
–
99.62
99.13
99.53
–
99.73
99.80
99.53
99.60
99.62
99.65
99.57
99.30
99.53
Table 1: Caption classification accuracy (%).
Rank
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Qwen3-VL
1
lighting suggests
overall impression
image depicts
detailed description
2
visible background
low resolution
image features
high contrast
3
light colored
slightly low angle
person wearing
central focus
4
appears taken
close slightly low
handwritten digit
main subject
5
photo taken
eye level view
lush green
depth field
6
lighting creates
high angle view
sunny day
shallow depth field
Table 2: Top distinctive TF-IDF phrases for each captioning model (generic appearance descriptors removed).
Figure 2: Word clouds of model-generated captions
Figure 3: Classification performance on generated images. The test accuracy of image classification on generated images is 49.85% with the SOTA model FLUX.1-schnell, while classification on a natural image dataset [ 20 ] of the same scale using the same network achieves 76.7%. Random guessing yields 33.3%.
Model
Total
Claude
Gemini
GPT
Original
Accuracy (%)
51.84
50.83
56.31
38.30
82.11
Table 6
Gemini
GPT
Claude
Total
keywords
95.37
94.48
88.73
92.86
images
46.67
41.34
41.56
43.22
Table 7
Model
Flux-schnell
SDXL
SD 2.1
SD 1.5
Accuracy (%)
46.05
45.69
44.76
41.67
Table 8
Text Transformation
Total
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Removing Markdown Format
99.71
99.73
99.62
99.77
Removing Special Characters
99.78
99.78
99.78
99.77
Shuffling Words
99.42
99.43
99.60
99.23
Shuffling Letters
34.49
0.00
100.00
3.48
Paraphrase 1 ( Qwen-2.5-1.5B-Instruct )
95.59
94.35
95.45
97.95
Paraphrase 1 ( Qwen-2.5-7B-Instruct )
95.90
92.68
97.73
97.30
Table 3: Total and per-class accuracy under different text modifications and paraphrases. Paraphrasing was performed with three distinct prompts on Qwen-2.5-1.5B-Instruct and Qwen-2.5-7B-Instruct to ensure robustness across rewording styles and model scales. Detailed information about prompts is provided in Appendix B .
Encoder
Claude
Gemini
GPT
Average
CLIP
94.47
95.1
92.85
94.14
T5
99.73
99.82
99.67
99.74
Table 4: Linear probe accuracy(%) on T2I text encoder embeddings. The stylistic signal is overwhelmingly preserved, not lost.
Model
Claude
Gemini
GPT
Average
Accuracy (%)
54.40
55.00
49.63
53.01
Table 11
Figure 4: Comparison of captions generated by Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o on the same images. Each row shows the original image on the left and, on the right, the captions generated by the three models and the images synthesized from those captions. The model outputs reveal several systematic failures across different attribute types. (i) In the first row, a simple descriptive color term, such as blue, without any accompanying texture specification, leads all models to produce images with broadly similar color–texture effects. However, none of the models reproduces the true, darker color in the original image. (ii) In the second column, even though captions include explicit view descriptions, the generated viewpoints remain inconsistent: a caption describing a high-angle view yields an eye-level rendering, while an eye-level description produces a low-angle output. (iii) In the third row, different color terms used to describe the acorn result in images that are still visually similar across models, and none of them match the actual color or appearance of the original image.
Figure 5: Detail-level rankings of both captions and generated images. Left: distribution of most-, moderately-, and least-detailed captions across the three models. Right: corresponding rankings assigned to the generated images.
Model
Basic (Total)
Nuanced (Total)
With Basic (%)
With Nuanced (%)
Basic (Average)
Nuanced (Average)
Color Vocabulary
Claude-3.5-Sonnet
88,797
23,235
92.36
43.96
2.96
0.77
Gemini-1.5-Pro
155,363
38,495
97.45
55.47
5.18
1.28
GPT-4o
62,843
10,186
81.01
22.95
2.09
0.34
Texture Vocabulary
Claude-3.5-Sonnet
27,009
32,034
67.24
52.91
0.90
1.07
Table 5: Lexical statistics of color and texture vocabulary in model-generated captions (30k per model). Metrics include total counts, percentage of captions containing at least one term, and average frequency per caption. Light green marks the highest value in each block, and light magenta marks the lowest.
Spatial
Subject
Guiding
Balance
Model
Layers
Focus
Elements
Symmetry
Claude-3.5-Sonnet
93.38
96.55
86.67
3.52
Gemini-1.5-Pro
90.70
90.09
66.99
0.54
GPT-4o
86.42
88.61
70.48
1.07
Table 6: Semantic composition analysis of captions. Values indicate the percentage (%) of captions meeting each criterion.
Figure 6: Per-class classification accuracy for generated images. Each bar shows how well images generated from captions of a given model can be attributed to their caption source.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: User study interface for caption and image attribution. Left: caption. Right: image. Top: examples. Bottom: test.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.
Text-to-image models have made tremendous progress and can produce high-quality images. However, a question which would be asked in any other discipline is where the frontier models fall short. We introduce GenOlympiad, a benchmark which provides the most challenging text prompts from nine capability axes for stress-testing frontier text-to-image models. The assessed capabilities include generating large quantities, performing deep reasoning, ensuring hard physical consistency, following complex rules and so on. The prompts are designed such that prompt alignment is visually verifiable (e.g., counting) instead of relying on subjectiveness (e.g., style). Based on this, we develop an automated evaluation framework which decomposes the prompt into easy and hard questions, and for most hard questions, draws help lines to ground evidence for accurate grading. We evaluate 20+ strong text-to-image models, including Nano Banana Pro, GPT-Image-2.5, Seedream-5.0 and so on. We report that even the best-performing T2I model (Nano Banana Pro) only achieves a strict score of 23.9 out of 100 and that coding models such as GPT-6 Astra have higher prompt alignment scores (but not as visually pleasing). We also derive insights in model evolution, alignment with Elo scores, and effectiveness of our grounding-based evaluation protocol. Code is available at https://github.com/GenOlympiad/GenOlympiad and the project page is at https://genolympiad.github.io/
Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. Project webpage: https://nsping13.github.io/contextless-TTI/
Nurit Spingarn, Noa Cohen, Tamar Rott Shaham +1
Technion – Israel Institute of Technology · MIT CSAIL