In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.
Figures & tables
Figure 1: Overview of our pipeline. Multiple MLLMs generate captions for the same image, and a text classifier reliably attributes each caption to its source model. These captions are then used to synthesize new images, on which an image classifier performs the same source-identification task. While text-based attribution is highly accurate, image-based attribution fails, which reveals a clear mismatch between caption-space and image-space model signatures.
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Qwen3-VL
Total
99.83
99.78
99.67
–
99.76
99.92
99.90
–
99.73
99.85
99.85
–
99.62
99.13
99.53
–
99.73
99.80
99.53
99.60
99.62
99.65
99.57
99.30
99.53
Table 1: Caption classification accuracy (%).
Rank
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Qwen3-VL
1
lighting suggests
overall impression
image depicts
detailed description
2
visible background
low resolution
image features
high contrast
3
light colored
slightly low angle
person wearing
central focus
4
appears taken
close slightly low
handwritten digit
main subject
5
photo taken
eye level view
lush green
depth field
6
lighting creates
high angle view
sunny day
shallow depth field
Table 2: Top distinctive TF-IDF phrases for each captioning model (generic appearance descriptors removed).
Figure 2: Word clouds of model-generated captions
Figure 3: Classification performance on generated images. The test accuracy of image classification on generated images is 49.85% with the SOTA model FLUX.1-schnell, while classification on a natural image dataset [ 20 ] of the same scale using the same network achieves 76.7%. Random guessing yields 33.3%.
Model
Total
Claude
Gemini
GPT
Original
Accuracy (%)
51.84
50.83
56.31
38.30
82.11
Table 6
Gemini
GPT
Claude
Total
keywords
95.37
94.48
88.73
92.86
images
46.67
41.34
41.56
43.22
Table 7
Model
Flux-schnell
SDXL
SD 2.1
SD 1.5
Accuracy (%)
46.05
45.69
44.76
41.67
Table 8
Text Transformation
Total
Claude-3.5-Sonnet
Gemini-1.5-Pro
GPT-4o
Removing Markdown Format
99.71
99.73
99.62
99.77
Removing Special Characters
99.78
99.78
99.78
99.77
Shuffling Words
99.42
99.43
99.60
99.23
Shuffling Letters
34.49
0.00
100.00
3.48
Paraphrase 1 ( Qwen-2.5-1.5B-Instruct )
95.59
94.35
95.45
97.95
Paraphrase 1 ( Qwen-2.5-7B-Instruct )
95.90
92.68
97.73
97.30
Table 3: Total and per-class accuracy under different text modifications and paraphrases. Paraphrasing was performed with three distinct prompts on Qwen-2.5-1.5B-Instruct and Qwen-2.5-7B-Instruct to ensure robustness across rewording styles and model scales. Detailed information about prompts is provided in Appendix B .
Encoder
Claude
Gemini
GPT
Average
CLIP
94.47
95.1
92.85
94.14
T5
99.73
99.82
99.67
99.74
Table 4: Linear probe accuracy(%) on T2I text encoder embeddings. The stylistic signal is overwhelmingly preserved, not lost.
Model
Claude
Gemini
GPT
Average
Accuracy (%)
54.40
55.00
49.63
53.01
Table 11
Figure 4: Comparison of captions generated by Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o on the same images. Each row shows the original image on the left and, on the right, the captions generated by the three models and the images synthesized from those captions. The model outputs reveal several systematic failures across different attribute types. (i) In the first row, a simple descriptive color term, such as blue, without any accompanying texture specification, leads all models to produce images with broadly similar color–texture effects. However, none of the models reproduces the true, darker color in the original image. (ii) In the second column, even though captions include explicit view descriptions, the generated viewpoints remain inconsistent: a caption describing a high-angle view yields an eye-level rendering, while an eye-level description produces a low-angle output. (iii) In the third row, different color terms used to describe the acorn result in images that are still visually similar across models, and none of them match the actual color or appearance of the original image.
Figure 5: Detail-level rankings of both captions and generated images. Left: distribution of most-, moderately-, and least-detailed captions across the three models. Right: corresponding rankings assigned to the generated images.
Model
Basic (Total)
Nuanced (Total)
With Basic (%)
With Nuanced (%)
Basic (Average)
Nuanced (Average)
Color Vocabulary
Claude-3.5-Sonnet
88,797
23,235
92.36
43.96
2.96
0.77
Gemini-1.5-Pro
155,363
38,495
97.45
55.47
5.18
1.28
GPT-4o
62,843
10,186
81.01
22.95
2.09
0.34
Texture Vocabulary
Claude-3.5-Sonnet
27,009
32,034
67.24
52.91
0.90
1.07
Table 5: Lexical statistics of color and texture vocabulary in model-generated captions (30k per model). Metrics include total counts, percentage of captions containing at least one term, and average frequency per caption. Light green marks the highest value in each block, and light magenta marks the lowest.
Spatial
Subject
Guiding
Balance
Model
Layers
Focus
Elements
Symmetry
Claude-3.5-Sonnet
93.38
96.55
86.67
3.52
Gemini-1.5-Pro
90.70
90.09
66.99
0.54
GPT-4o
86.42
88.61
70.48
1.07
Table 6: Semantic composition analysis of captions. Values indicate the percentage (%) of captions meeting each criterion.
Figure 6: Per-class classification accuracy for generated images. Each bar shows how well images generated from captions of a given model can be attributed to their caption source.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 7: User study interface for caption and image attribution. Left: caption. Right: image. Top: examples. Bottom: test.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.