ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · China Academy of Art, Hangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Abstract
Text-to-image (T2I) models have advanced considerably in generating high-quality images from textual descriptions. However, their ability to associate colors with concepts remains largely constrained to explicit color names or codes, while their capacity to handle implicit concepts, such as emotions and visual states, remains underexplored. To address this gap, we introduce ColorConceptBench, an expert-annotated benchmark that systematically evaluates color-concept associations through probabilistic color distributions. ColorConceptBench moves beyond explicit color specifications by examining how models interpret 1,281 implicit color concepts, grounded in 6,584 human annotations. Our evaluation of nine leading T2I models reveals that performance varies substantially across semantic categories, and models exhibit a significant lack of sensitivity to abstract semantics. These limitations persist even when applying classifier-free guidance scaling at inference time, suggesting that achieving human-like color understanding demands a shift in how models learn and represent implicit semantic meaning.
Figures & tables
| Benchmark | Scale | Data Format | Data Collection | Focus |
|---|---|---|---|---|
| GenEval [ 11 ] | 553 | Text prompts | Manual curation | Compositionality |
| T2I-CompBench++ [ 20 ] | 8,000 | Text prompts | Manual curation | Compositionality |
| DPG-Bench [ 19 ] | 1,065 | Text prompts | Manual curation | Prompt Adherence |
| Commonsense-T2I [ 10 ] | 1,000 | Adversarial prompt pairs | Expert annotation | Commonsense Reasoning |
| WISE [ 32 ] | 1,000 | Text prompts | Manual curation | Commonsense Reasoning |
| MMMG [ 30 ] | 4,456 | Image-prompt pairs + KGs | Expert annotation | Disciplinary Knowledge |
| Method | Original | Visual state | Emotional | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Natural | Clipart | Natural | Clipart | Natural | Clipart | |||||||||||||
| PCC | EMD | ED | PCC | EMD | ED | PCC | EMD | ED | PCC | EMD | ED | PCC | EMD | ED | PCC | EMD | ED | |
| Flux.1-dev | 0.369 | 29.165 | 0.621 | 0.391 | 26.915 | 0.578 | 0.378 | 28.413 | 0.596 | 0.337 | 28.944 | 0.501 | 0.407 | 28.718 | 0.771 | 0.344 | 27.959 | 0.451 |
| OmniGen | 0.348 | 30.866 | 0.575 | 0.363 | 31.413 | 0.581 | 0.319 | 32.658 | 0.974 | 0.278 | 33.840 | 0.761 | 0.342 | 32.201 | 1.258 | 0.290 | 33.319 | 0.925 |
| OmniGen2 | 0.355 | 30.066 | 0.648 | 0.321 | 36.349 | 0.845 | 0.320 | 31.572 | 0.960 | 0.224 | 39.130 | 1.463 | 0.348 | 30.346 | 1.090 | 0.261 | 35.260 | 1.447 |
| PixArt- | 0.351 | 30.867 | 0.596 | 0.360 | 29.663 | 0.607 | 0.360 | 29.747 | 0.577 | 0.333 | 30.554 | 0.536 | 0.389 | 30.032 | 0.763 | 0.425 | 26.070 | 0.528 |
| Type | Metric | Kendall’s Tau | Spearman | Agreement |
|---|---|---|---|---|
| Probabilistic | PCC | 0.230 | 0.316 | 63.65% |
| EMD | 0.276 | 0.378 | 68.37% | |
| ED | 0.173 | 0.235 | 47.24% | |
| Deterministic | DCA | 0.118 | 0.136 | 11.94% |
| Hue | 0.162 | 0.219 | 43.44% |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Lexicon Link |
|---|---|
| Fruit & Vegetables | https://www.words-to-use.com/words/fruits-vegetables/ |
| Water Body | https://www.yourdictionary.com/articles/water-words-descriptive-writing |
| Weather & Sky | https://www.writerswrite.co.za/words-to-describe-weather/ |
| Floral | https://www.slowflowerspodcast.com/wp-content/uploads/Slow-Flowers_Creative_Workshop_Floral-Adjectives_Nouns_Verbs.pdf |
| Terrain | https://adjectives-for.com/terrain |
| https://simplicable.com/descriptive-words/words-to-describe-nature |
| Expert Votes | Interpretation | Image Count |
|---|---|---|
| 3 Yes / 0 No | Unanimous Consistent | 399 |
| 0 Yes / 3 No | Unanimous Inconsistent | 4 |
| 2 Yes / 1 No | Majority Consistent | 127 |
| 1 Yes / 2 No | Majority Inconsistent | 32 |
| Total | 562 |
| EMD | PCC | ||||
|---|---|---|---|---|---|
| Rank | Model | Human-derived Ref. | Internet-derived Ref. | Human-derived Ref. | Internet-derived Ref. |
| 1 | SD XL | 28.507 [26.574, 30.441] | 18.652 [16.960, 20.344] | 0.341 [0.290, 0.392] | 0.637 [0.585, 0.689] |
| 2 | Flux | 29.055 [26.903, 31.208] | 20.638 [18.956, 22.320] | 0.368 [0.316, 0.420] | 0.603 [0.551, 0.655] |
| 3 | Sana | 29.192 [27.108, 31.276] | 21.232 [19.111, 23.354] | 0.372 [0.319, 0.425] | 0.597 [0.541, 0.652] |
| 4 | SD 3 | 30.394 [28.185, 32.603] | 21.306 [19.355, 23.257] | 0.320 [0.270, 0.369] | 0.568 [0.509, 0.626] |
| 5 | Pixart- | 31.331 [28.986, 33.676] | 22.005 [19.922, 24.087] | 0.341 [0.289, 0.393] | 0.597 [0.543, 0.651] |
| EMD | PCC | ||||
|---|---|---|---|---|---|
| Rank | Model | Human-derived Ref. | Internet-derived Ref. | Human-derived Ref. | Internet-derived Ref. |
| 1 | SD XL | 27.689 [25.887, 29.491] | 21.403 [19.497, 23.309] | 0.351 [0.300, 0.402] | 0.570 [0.518, 0.623] |
| 2 | Pixart- | 32.300 [29.919, 34.680] | 26.067 [23.580, 28.554] | 0.329 [0.278, 0.380] | 0.506 [0.450, 0.562] |
| 3 | Flux | 29.429 [27.310, 31.547] | 26.255 [24.151, 28.359] | 0.342 [0.288, 0.396] | 0.464 [0.411, 0.517] |
| 4 | Sana | 29.358 [26.997, 31.718] | 26.481 [23.674, 29.289] | 0.394 [0.346, 0.442] | 0.491 [0.437, 0.544] |
| 5 | SD 3.5 | 32.499 [30.558, 34.440] | 27.039 [24.677, 29.401] | 0.301 [0.246, 0.355] | 0.482 [0.429, 0.536] |
| Method | Style | |
|---|---|---|
| Clipart | Natural | |
| Flux | 0.677 | 0.631 |
| OmniGen | 0.789 | 0.896 |
| OmniGen2 | 1.027 | 0.800 |
| Pixart- | 0.742 | 0.688 |
| Qwen-Image | 0.861 | 1.047 |
| Method | Modifier | |
|---|---|---|
| Visual State | Emotional | |
| Flux | 0.656 | 0.645 |
| OmniGen | 0.860 | 0.729 |
| OmniGen2 | 0.904 | 0.948 |
| Pixart- | 0.709 | 0.752 |
| Qwen-Image | 0.974 | 0.818 |
| Method | Style | ||
|---|---|---|---|
| Clipart | Natural | Average | |
| Emotional | 0.773 | 0.659 | 0.716 |
| Visual State | 0.800 | 0.739 | 0.769 |
| Method | Original | Visual state | Emotional | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Natural | Clipart | Natural | Clipart | Natural | Clipart | |||||||
| DCA | Hue | DCA | Hue | DCA | Hue | DCA | Hue | DCA | Hue | DCA | Hue | |
| Flux.1-dev | 0.187 | 33.952 | 0.201 | 27.987 | 0.172 | 38.647 | 0.170 | 36.656 | 0.233 | 45.881 | 0.120 | 58.490 |
| OmniGen | 0.178 | 35.295 | 0.168 | 32.443 | 0.140 | 41.013 | 0.117 | 41.649 | 0.180 | 53.326 | 0.053 | 48.599 |
| OmniGen2 | 0.173 | 31.798 | 0.178 | 30.412 | 0.147 | 38.903 | 0.109 | 35.708 | 0.135 | 53.816 | 0.098 | 50.383 |
| PixArt- | 0.178 | 32.890 | 0.164 | 32.512 | 0.165 | 42.138 | 0.155 | 38.120 | 0.233 | 42.620 | 0.203 | 44.837 |
| Clipart | Natural | |||
|---|---|---|---|---|
| Model | ||||
| Flux | 2.423 | -0.057 | -2.081 | 0.022 |
| OmniGen2 | 4.232 | -0.100 | 1.978 | -0.045 |
| OmniGen | 3.759 | -0.083 | 0.847 | -0.011 |
| Pixart- | 2.580 | -0.057 | -2.076 | 0.018 |
| Qwen-Image | 1.118 | -0.059 | -0.763 | 0.012 |
| Clipart | Natural | |||
|---|---|---|---|---|
| Model | ||||
| Flux | 0.276 | 0.025 | -3.413 | 0.089 |
| OmniGen2 | 3.444 | -0.024 | -0.241 | 0.032 |
| OmniGen | 2.436 | 0.021 | -4.716 | 0.105 |
| Pixart- | -1.112 | 0.099 | -5.802 | 0.114 |
| Qwen-Image | 1.424 | 0.017 | 0.283 | 0.064 |
| Method | Q1 | Median | Q3 |
|---|---|---|---|
| Flux | 0.346 | 0.542 | 0.855 |
| OmniGen | 0.446 | 0.720 | 1.108 |
| OmniGen2 | 0.505 | 0.781 | 1.210 |
| Pixart- | 0.380 | 0.614 | 0.938 |
| Qwen-Image | 0.551 | 0.856 | 1.237 |
| SD 3 | 0.393 | 0.612 | 0.897 |
| Guidance Scale | |||||||
|---|---|---|---|---|---|---|---|
| Method | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| Flux | |||||||
| OmniGen | |||||||
| OmniGen2 | |||||||
| Pixart- | |||||||
| Qwen-Image | |||||||
| Guidance Scale | |||||||
|---|---|---|---|---|---|---|---|
| Method | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| Flux | |||||||
| OmniGen | |||||||
| OmniGen2 | |||||||
| Pixart- | |||||||
| Qwen-Image | |||||||
| Style | Metric | Friedman | Significant pairs | ||
|---|---|---|---|---|---|
| Natural | EMD | 1271 | 559.17 | 29/36 | |
| Natural | PCC | 1271 | 300.07 | 27/36 | |
| Clipart | EMD | 1274 | 1737.65 | 34/36 | |
| Clipart | PCC | 1274 | 717.90 | 27/36 |
| Clipart | Natural | |||
|---|---|---|---|---|
| Model | EMD | PCC | EMD | PCC |
| Flux | ||||
| OmniGen | ||||
| OmniGen2 | ||||
| Pixart- | ||||
| Qwen-Image | ||||
| Clipart | Natural | |||
|---|---|---|---|---|
| Model | EMD | PCC | EMD | PCC |
| Flux | ||||
| OmniGen | ||||
| OmniGen2 | ||||
| Pixart- | ||||
| Qwen-Image | ||||
| Clipart | Natural | |||
|---|---|---|---|---|
| Model | EMD | PCC | EMD | PCC |
| Flux | ||||
| OmniGen | ||||
| OmniGen2 | ||||
| Pixart- | ||||
| Qwen-Image | ||||
| Model | Original EMD | Original PCC | Visual EMD | Visual PCC | Emotional EMD | Emotional PCC |
|---|---|---|---|---|---|---|
| Flux.1-dev | 29.555 | 0.366 | 28.114 | 0.373 | 29.460 | 0.344 |
| OmniGen | 30.838 | 0.338 | 35.046 | 0.305 | 36.085 | 0.271 |
| OmniGen2 | 31.710 | 0.349 | 34.527 | 0.298 | 32.105 | 0.309 |
| PixArt- | 31.221 | 0.323 | 30.543 | 0.369 | 36.185 | 0.276 |
| Qwen-Image | 32.058 | 0.301 | 31.458 | 0.319 | 27.408 | 0.356 |
| SD3 | 29.923 | 0.307 | 31.112 | 0.321 | 27.750 | 0.397 |
| Model | Original EMD | Original PCC | Visual EMD | Visual PCC | Emotional EMD | Emotional PCC |
|---|---|---|---|---|---|---|
| Flux | 28.585 | 0.367 | 30.890 | 0.322 | 25.585 | 0.289 |
| Omni | 31.214 | 0.383 | 37.470 | 0.265 | 33.650 | 0.345 |
| Omni2 | 37.022 | 0.346 | 42.367 | 0.218 | 30.005 | 0.420 |
| PixArt | 30.998 | 0.324 | 35.207 | 0.304 | 23.207 | 0.499 |
| Qwen | 32.026 | 0.298 | 33.052 | 0.246 | 27.744 | 0.311 |
| SD3 | 30.401 | 0.330 | 35.218 | 0.258 | 27.069 | 0.345 |
| Id | Category | Concept | Visual state modifier | Emotional modifier |
|---|---|---|---|---|
| 1 | animal | dog | ||
| 2 | animal | bird | ||
| 3 | animal | horse | ||
| 4 | animal | chicken | ||
| 5 | animal | bear | ||
| 6 | animal | cat |
T2I Generation Evaluation
Text-to-image generation evaluation measures how accurately, safely, and consistently image models turn prompts into images.
See all 182 papers