ACVA-Bench: Beyond Prompt Following in Abstract Concept Visualization Ability for Text-to-Image Models
Authors: Mengyan Wang, Jiyuan An, Yining Liu, YaoWen Zhang, Liner Yang ang Erhong Yang
Organizations: National Language Resources Monitoring and Research Center for Print Media, Beijing Language and Culture University, China · School of Information Science, Beijing Language and Culture University, China
Text-to-image (T2I) models have achieved remarkable progress in image quality and instruction following, prompting growing research interest in their capacity for deep understanding that goes beyond literal prompt matching. However, without an explicit visual scheme provided, it remains unclear and has not been systematically evaluated whether T2I models can autonomously translate abstract concepts lacking a fixed visual referent, such as "loyalty," into visual representations that are accurate, diverse, and distinguishable from closely related concepts. We define this capability as Abstract Concept Visualization Ability (ACVA) and introduce ACVA-Bench, a category-balanced benchmark grounded in psycholinguistic norms of abstract concepts, which examines models' ACVA along three dimensions semantic sufficiency, representational diversity, and conceptual distinctiveness under a low-intervention setting that withholds concrete objects, scenes, or visual metaphors. Since abstract concepts lack a unique reference image, conventional evaluation metrics that rely on reference-image matching are difficult to apply. To address this, we propose a fine-grained visual-semantic evaluation method constrained by concept semantics, which decomposes concept semantics into independently verifiable visual-semantic questions to assess the semantic representation quality of generated images, with its validity further confirmed by human evaluation. Benchmarking results on mainstream T2I models show that these models rely on a small set of prototypical visual patterns, leaving room for improvement in representational diversity and conceptual distinctiveness for abstract concepts, thereby providing an interpretable and reproducible evaluation foundation for future research.
Figures & tables
Figure 1: Motivation. Concrete concepts have fixed visual referents, whereas abstract concepts require models to autonomously select visual entities, relations, and metaphors, often resulting in divergent and incomplete representations.
Figure 2: Overview of ACVA-Bench: concept sampling, rubric construction, low-intervention generation, and three evaluation axes, with human quality control.
Figure 3: Rubric construction pipeline from dictionary definitions to human-reviewed visual judgment questions. Constraints encode necessary semantic conditions rather than fixed objects or symbols; weights are human-calibrated.
Model
Mean
SD
Pass (%)
GPT-Image-2
0.874
0.191
90.4%
Gemini-3.1-Flash
0.781
0.288
79.6%
Qwen-Image-2.0
0.556
0.378
53.6%
GLM-Image
0.531
0.373
48.7%
Qwen-Image
0.170
0.301
12.7%
NextStep-1.1
0.155
0.278
10.2%
Table 1: Semantic adequacy by model (weighted visual-semantic rubric score; n=450 per model). Pass denotes the percentage scoring at least 0.6.
Figure 4: Adequacy by model and cognitive category (75 images per cell). Interpersonal communication has the lowest mean across models.
Model
Dimg
Dtext
H
Pseudo
Low
Qwen-Image
0.297
0.280
1.15
10.7%
26
NextStep-1.1
0.286
0.198
0.97
31.3%
37
GLM-Image
0.274
0.262
1.28
14.7%
19
Gemini-3.1-Flash
0.246
0.209
1.01
20.0%
49
Qwen-Image-2.0
0.228
0.194
0.92
12.0%
65
GPT-Image-2
0.173
0.138
0.52
6.0%
112
Table 2: Diversity metrics by model (900 model × concept groups in total; 150 per model). Dimg / Dtext are mean pairwise within-group image/metaphor-embedding distances, H is metaphor entropy, Pseudo is the pseudo-diversity rate, and Low is the number of groups falling in the low-image-distance + low-metaphor-diversity quadrant.
Figure 5: Normalized image and metaphor-text distances for all 900 groups (left) and 440 adequacy-qualified groups (right). Dashed lines show the median splits used by the diagnostic.
Model
Img LOO
MMD
Txt LOO
2AFC
NextStep-1.1
0.883
0.440
0.850
0.728
GPT-Image-2
0.861
0.454
0.878
0.821
Qwen-Image-2.0
0.717
0.347
0.800
0.761
GLM-Image
0.650
0.288
0.733
0.839
Gemini-3.1-Flash
0.611
0.271
0.772
0.833
Qwen-Image
0.517
0.183
0.689
0.906
Table 3: Near-synonym distinctiveness results (30 pairs × 6 models; chance level 0.5). Img/Txt LOO denote leave-one-out nearest-centroid classification accuracy in image/text embedding space; VLM 2AFC is the primary perceptual-side metric.
Figure 6: Core failure modes. (a) Three GPT-Image-2 generations for marriage repeat one prototype. (b) Oath/promise shares visual motifs (pair-level VLM accuracy 0.47). (c) Marriage/family has distinct visual carriers (1.00). Images are from GPT-Image-2; accuracies summarize all six models.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Canonical category
Figure shorthand
Example concepts
Physical, spatiotemporal, and quantitative
Physical / Spatiotemporal
color, space, distance, weather
Emotion, sensation, and experience
Emotion / Experience
happiness, shame, charm, nostalgia
Quality, mentality, and belief
Qualities & Ideas
loyalty, freedom, belief, faith
Social-situation
Social Situations
marriage, family, conflict, violence
Interpersonal-communication
Interpersonal Communication
disagreement, excuse, criticism, explanation
Terminological, institutional, and epistemic
Institutional / Cognitive
economy, capital, system, structure
Appendix
Table 4: Canonical category names and example concepts. Each category contains 25 concepts. The displayed column order is also used in Table 7 .
Illustrative visual-semantic question
Priority
Does the image depict an ongoing, temporally extended relationship or bond, rather than a single isolated event?
high
Does one party show sustained support or allegiance toward another, rather than mere physical proximity or companionship?
highest
Is there a directional or asymmetric relation (e.g., one figure oriented toward, protecting, or standing by another) rather than a neutral symmetric pairing?
medium
Is attachment or bonding signified through a container/link schema (e.g., linked hands, rings, or an enclosing gesture) rather than incidental adjacency?
medium
Does the image avoid depicting betrayal, abandonment, or breach, which would contradict the target sense?
medium
Appendix
Table 5: Illustrative questions for loyalty . Priorities reproduce the schematic example; actual scores use the numeric weights established by human review (Section 3.3 ).
Rank
Concept pair (English glosses)
Mean VLM
Mean LOO
1
oath / promise
0.472
0.528
2
system / structure
0.556
0.750
3
custom / folklore
0.583
0.694
4
perseverance / resolve
0.694
0.556
5
belief / faith
0.722
0.528
6
anger / hatred
0.722
0.806
Appendix
Table 6: All 30 near-synonym pairs, ordered by mean VLM 2AFC accuracy across six models. LOO denotes image-embedding leave-one-out nearest-centroid accuracy.
Model
Phys.
Emot.
Qual.
Social
Interp.
Term.
GPT-Image-2
0.907
0.913
0.872
0.891
0.830
0.834
Gemini-3.1-Flash
0.797
0.803
0.782
0.771
0.716
0.816
Qwen-Image-2.0
0.712
0.629
0.580
0.562
0.254
0.597
GLM-Image
0.724
0.536
0.517
0.565
0.305
0.540
Qwen-Image
0.325
0.232
0.137
0.161
0.061
0.102
NextStep-1.1
0.379
0.137
0.139
0.075
0.020
0.178
Appendix
Table 7: Semantic adequacy by model and category, rounded to three decimals; each cell averages 75 images. Columns follow Table 4 . Category means use the underlying unrounded values.
Figure 7: Mean adequacy by cognitive category. The dashed line is the overall mean (0.511); canonical names are in Table 4 .
Figure 8: Semantic pass rate versus diversity across six models. Axes show Dimg and metaphor entropy H ; marker size and color encode the pass rate in Table 1 .
Figure 9: MMD-based separability for 30 pairs and six models (RBF kernel), sorted by mean MMD. English labels gloss the Chinese concept pairs. This figure reports MMD, not perceptual 2AFC accuracy.
Figure 10: Imageability versus mean semantic adequacy for 150 concepts, colored by category. The dashed line is an OLS fit; the reported association is Spearman ρ=0.215 .
Figure 11: Adequacy versus form diversity for 900 model–concept groups, colored by model ( ρ=−0.286 ).
Figure 12: Pair-level adequacy versus VLM 2AFC accuracy over 180 model–pair results ( ρ=−0.044 , p=0.55 ).
Model
Mean [95% CI]
GPT-Image-2
0.874 [0.848, 0.899]
Gemini-3.1-Flash
0.781 [0.742, 0.818]
Qwen-Image-2.0
0.556 [0.504, 0.608]
GLM-Image
0.531 [0.483, 0.579]
Qwen-Image
0.170 [0.130, 0.213]
NextStep-1.1
0.155 [0.116, 0.196]
Appendix
Table 8: Cluster-bootstrap intervals for mean adequacy (150 concept clusters; 5,000 resamples).
Figure 13: Model adequacy with 95% cluster-bootstrap intervals. Pairwise significance is assessed using difference tests in Appendix D.3 .
Model
Unweighted
Weighted
Reported Δ
GPT-Image-2
0.871
0.874
+0.004
Gemini-3.1-Flash
0.776
0.781
+0.005
Qwen-Image-2.0
0.562
0.556
-0.006
GLM-Image
0.535
0.531
-0.004
Qwen-Image
0.170
0.170
-0.000
NextStep-1.1
0.162
0.155
-0.007
Appendix
Table 9: Equal-weight versus human-calibrated mean adequacy. Reported deltas can differ from subtraction of the displayed rounded means.
Comparison
n
Spearman ρ
Exact
± 1
Inter-annotator (mean)
90
0.622
0.696
0.970
Auto vs. pooled human
270
0.362
0.463
0.911
Appendix
Table 10: Human validation of the weighted semantic score on 90 images. Auto = Qwen3.6-27B rubric scoring; inter-annotator Fleiss κ=0.515 ; Exact = exact agreement; ± 1 = agreement within one rating bin.
Figure 14: Three independent GPT-Image-2 generations for marriage and freedom , with one contrasting model per row. Similar schemas across repeated samples illustrate prototype reliance.
Figure 15: Cross-model comparison for (a) oath/promise and (b) marriage/family . Three models illustrate recurring motif overlap in the harder pair and more distinct visual realizations in the easier pair.
Figure 16: Mean semantic adequacy and pass rate ( ≥ 0.6 weighted score) by model.