KnowVis: Dual-View Evaluation and Diagnosis of World-Knowledge Grounding in Text-to-Image Models
Organizations: KlingAI Research
Abstract
Text-to-image (T2I) models have made substantial progress in visual realism, aesthetic quality, and instruction following. However, real-world prompts often go beyond explicit visual descriptions and require implicit facts, structured knowledge, and domain-specific commonsense. Existing evaluations mainly focus on explicit prompt-to-image semantic alignment or specific knowledge subdomains, lacking unified evaluation and diagnosis of world-knowledge grounding in T2I models. We introduce KnowVis for dual-view evaluation and diagnosis of world-knowledge grounding in T2I generation. Built on a hierarchical knowledge taxonomy with case-level annotations, KnowVis combines two complementary benchmark views with a structured MLLM-based VQA judge protocol. The Skill-Tree Benchmark supports fine-grained atomic knowledge evaluation, while the Generalized Benchmark examines open-ended knowledge composition and expression. The protocol uses Required VQAs to assess atomic knowledge correctness in the Skill-Tree Benchmark, and combines Required, Optional, and dynamically discovered knowledge expressions with human-judge collaborative evaluation in the Generalized Benchmark. We validate the protocol's reliability on the Skill-Tree validation split, enabling scalable evaluation and diagnostic error attribution. Experiments show that KnowVis reveals notable gaps in world-knowledge grounding across existing T2I models and provides structured diagnostic signals for knowledge-intensive image generation.
Figures & tables
| VQA-level human–judge agreement | Ranking consistency | |||||
| Judge setting | Acc. | Prec. | Rec. | Kendall’s | Spearman’s | |
| Gemini 3.1 Pro Preview (original) | 94.51 | 92.07 | 80.22 | 0.8236 | – | – |
| Gemini 3.1 Pro Preview (rerun) | 91.74 | 83.63 | 74.44 | 0.7366 | 0.8222 | 0.9273 |
| Gemini 3.5 Flash | 92.21 | 86.74 | 73.41 | 0.7475 | 0.8222 | 0.9394 |
| GPT 5.5 | 91.33 | 86.03 | 69.11 | 0.7140 | 0.8222 | 0.9394 |
| Slice | Category | Rewritten | Default | |
| Overall | ALL | 0.408 | 0.271 | +0.137 |
| Knowledge operation | Explicit Knowledge Recall | 0.432 | 0.296 | +0.136 |
| Implicit Knowledge Recall | 0.667 | 0.333 | +0.334 | |
| Knowledge Reasoning | 0.263 | 0.132 | +0.131 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Organizational Unit | Definition | Primary Role |
| L1 | Domain | The broadest knowledge category, defining the top-level thematic scope of the taxonomy. Each L1 domain groups a set of related subjects that share a common epistemic nature (e.g., natural sciences, human civilization). | Supports overall coverage analysis and cross-domain capability comparison across the benchmark. |
| L2 | Subject | A stable disciplinary subdivision within an L1 domain. Each L2 subject corresponds to a well-established field of knowledge (e.g., architecture, biology), providing the structural backbone for organizing lower-level branches. | Ensuring a smooth transition from abstract domains to concrete branches. |
| L3 | Branch | A specific knowledge direction within an L2 subject. Concrete enough to define a visual task scenario (e.g., architectural styles, animal anatomy). | Serves as the basis for constructing the Generalized Benchmark. |
| L4 | Knowledge Point | The minimal, atomic unit of knowledge. Each L4 knowledge point has a clearly defined scope, a single verifiable claim, and a stable correctness criterion. | Directly used as the basic unit for Skill-Tree Benchmark construction, MLLM-as-a-judge evaluation, and attribution analysis. |
| Benchmarks | Taxonomy | Dataset Structure | Evaluation Protocol | Attribution | ||||||||
| N | F | H | A | R | Atomic | Generalized | Form. | MLLM. | Human Val. | Diff. & Edu. | Text/Anno. | |
| WISE [ 13 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | Score-based | ✓ | ✓ | ✗ | ✗ |
| ABP [ 15 ] | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | Score-based | ✓ | ✓ | ✗ | ✗ |
| T2I-ReasonBench [ 26 ] | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | Score-based | ✓ | ✓ | ✗ | ✗ |
| GIR-Bench (T2I) [ 33 ] | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | Score-based | ✗ | ✓ | ✗ | ✗ |
| UniGenBench++ [ 34 ] | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | Score-based | ✓ | ✗ | ✗ | ✗ |
| L1 Domain | Skill-Tree | Generalized | ||
| Count | Percent | Count | Percent | |
| Natural Science | 841 | 51.8% | 224 | 40.7% |
| Human Civilization and Society | 489 | 30.1% | 180 | 32.7% |
| Formal Science | 292 | 18.0% | 71 | 12.9% |
| Arts and Media | - | - | 50 | 9.1% |
| Real-time Dynamic Events | - | - | 25 | 4.5% |
| Attribute | Sub-label | Count | Percent |
| Education level | Primary school | 166 | 10.2% |
| Middle school | 715 | 44.1% | |
| High school | 435 | 26.8% | |
| University | 99 | 6.1% | |
| Professional domain | 207 | 12.8% | |
| Knowledge operation type | Explicit Knowledge Recall | 1,322 | 81.5% |
| Attribute | Sub-label | Count | Percent |
| Difficulty | Low | 293 | 53.3% |
| Medium | 176 | 32.0% | |
| High | 81 | 14.7% | |
| Education level | Primary school | 27 | 4.9% |
| Middle school | 170 | 30.9% | |
| High school | 94 | 17.1% |
| Education Level | Low | Medium | High | Total |
| Primary school | 18 | 9 | 0 | 27 |
| Middle school | 120 | 40 | 10 | 170 |
| High school | 52 | 24 | 18 | 94 |
| University | 5 | 1 | 4 | 10 |
| Professional domain | 98 | 102 | 49 | 249 |
| Total | 293 | 176 | 81 | 550 |
| Judge Type | Definition | Affected Metrics |
| REQUIRED | Mandatory items. Elements explicitly required by the original prompt, or conditions that are physically absolute and implicitly necessary to realize the described scene or knowledge point in the physical world. Example: a prompt requesting "a submarine underwater" must include an underwater environment; a prompt requesting "an apple falling" must depict the apple airborne with a downward trajectory. | Knowledge Correctness |
| OPTIONAL | Optional items. Elements not explicitly required by the original prompt, but highly likely to be spontaneously generated by T2I models when processing knowledge-intensive tasks—most typically: additional textual annotations, arrows, auxiliary diagrams, or explanatory labels. The judgment logic for this type must follow a "penalize only if attempted incorrectly" principle: items are silently marked NA if absent, and marked FAIL only if generated incorrectly. Common failure cases: a model asked to plot a function curve that adds axis-point annotations with incorrect coordinates; a model depicting a historical scene whose surrounding environment is anachronistic. | Knowledge Richness |
| ADDITIONAL | Supplementary items. Like OPTIONAL , but derived from multimodal understanding of the actually generated image: additional elements spontaneously produced by the T2I model are identified, converted into verification checkpoints that define their correct appearance, and appended as supplementary VQA items. | None |
| Kind | Component Type | Definition |
| Entity Centric | ENTITY | Entity existence. Whether an entity composed of general, common-level concepts is present in the image — including specific persons, objects, phenomena, scenes, and backgrounds. Scene text and graphics attached to entities, as well as the overall style and genre of the image, are also included in this type. |
| ENTITY_TRAIT | Entity traits. Objective, intrinsic properties of knowledge-intensive entities that remain invariant across viewpoints, external environments, and variant subtypes. These represent the characteristics that a T2I model must correctly and fully render — beyond merely generating a generic entity — in order for the depiction to be recognizable as that specific knowledge-intensive entity. Accordingly, when an entity is classified as knowledge-intensive, an ENTITY_TRAIT VQA item is added alongside the corresponding generic ENTITY existence item. | |
| ATTRIBUTE_BINDING | Attribute binding. An entity’s color, shape, size, material, state, and similar properties. | |
| ENTITY_RELATION | Inter-entity relations. Spatial positioning, physical interaction, and temporal ordering among entities. | |
| Annotation Centric | ANNOTATION | Annotation content and form. Virtual textual or graphical annotations detached from entities, including text of all types, symbols, formulas, charts, figures, and diagrams. |
| ANNOTATION_BINDING | Entity–annotation binding. The directional or positional relationship between an annotation and its associated entity. |
| Vote pattern | Count | Percentage | Majority PASS | Majority FAIL |
| 5–0 | 238 | 82.64% | 190 | 48 |
| 4–1 | 29 | 10.07% | 23 | 6 |
| 3–2 | 21 | 7.29% | 12 | 9 |
| Total | 288 | 100.00% | 225 | 63 |
| Dimension | Evaluation Dimension | Definition |
| World-Knowledge Grounding | Knowledge Correctness | The degree to which core knowledge expressions are rendered correctly, reflecting the model’s understanding of the relevant world knowledge. |
| Knowledge Richness | The extent to which the model supplements core knowledge points with additional topically relevant knowledge, reflecting the depth and breadth of its world-knowledge understanding. | |
| Semantic | Non-Knowledge Visual Instruction Following | The degree to which visual requirements specified in the prompt—beyond knowledge correctness and richness—are fulfilled; i.e., the semantic alignment between the image and the instruction. |
| Visual | Aesthetics | An assessment of the aesthetic quality of the generated image, including lighting, color, composition, and other aesthetic factors, reflecting the model’s capacity for visually appealing generation. |
| Image Quality | An assessment of visual quality, including sharpness, structural integrity, detail fidelity, and realism. Factual errors do not, by themselves, lower this score; a lower score requires an independently observable visual defect, such as blurring, structural distortions, or geometric interpenetration. | |
| Holistic | Overall Satisfaction | A holistic judgment representing the ultimate user acceptance of the generated image. Rather than a linear weighted average of other dimensions, it reflects a comprehensive, task-context-dependent utility assessment that accounts for complex cross-dimensional interactions |
| Score | Image Quality | Aesthetics | Overall Satisfaction |
| 1 Bad | Severe blur, distortion, fragmentation, or structural collapse makes the main subjects or essential visual information unrecognizable. The image cannot support basic comprehension. | Severely disorganized composition, conflicting colors, or incoherent visual treatment. The image lacks a clear hierarchy and produces a strongly unfavorable visual impression. | The task is fundamentally unfulfilled. Severe core-knowledge errors, missing essential content, or overwhelming visual defects make the result unusable. |
| 2 Poor | Subjects remain recognizable, but substantial deformation, misalignment, interpenetration, or poorly rendered details seriously impair visual clarity and usability. | Weak composition or an unclear focal point, with distracting clutter, discordant colors, or inconsistent styling. These issues substantially impair visual appeal. | The result falls substantially short of the task. Major knowledge errors, missing supporting information, instruction violations, or poor visual execution substantially limit its usefulness. |
| 3 Fair | Main subjects and structures are discernible and broadly intact. Noticeable local artifacts, blur, repetition, or weak details remain, but the image is visually usable. | Acceptable composition and generally compatible colors and styling, but limited hierarchy, balance, or refinement. The overall appearance is adequate but unremarkable. | The result is usable but only partially fulfills the task. Knowledge coverage, instruction following, or visual presentation has noticeable limitations that reduce its overall usefulness. |
| 4 Good | Clear subjects and most details, with coherent structures and spatial relations. Only minor local artifacts or simplifications remain, without materially impairing comprehension. | Well-organized composition, a clear focal point, and harmonious visual treatment. Minor weaknesses in spacing, balance, or refinement do not undermine the favorable overall impression. | The task is largely fulfilled, with substantially correct core knowledge, sufficient supporting content, and good visual execution. Minor omissions or imperfections remain. |
| 5 Excellent | Crisp, complete rendering with stable details and coherent geometry. Textures and lighting are appropriate to the visual form, with no evident distracting artifacts. | Refined composition, clear visual hierarchy, and cohesive colors and styling. The presentation is highly effective and visually appealing for the intended task. | The task is fulfilled exceptionally well. Core knowledge is accurate, relevant supporting content is comprehensive, and instructions are faithfully followed, with strong visual execution and high overall usefulness. |
| Initial rating pattern | Score range | Percentage |
| Unanimous agreement | 0 | 57.02% |
| Adjacent-level disagreement | 1 | 30.17% |
| Larger disagreement | 12.81% |
| Benchmark View | Reporting Format | Purpose |
| Skill-Tree | L1/L2/L3/L4 hierarchical table | Shows accuracy across domains, branches, and L4 atomic knowledge points. |
| VQA component heatmap | Identifies dominant failure types, such as entity, relation, annotation, or binding errors. | |
| L4-under-L3 heatmap | Reveals which L4 atomic knowledge points fail within the same L3 branch. | |
| Annotation-related failure table | Measures whether text, formulas, arrows, labels, or callouts form a bottleneck for knowledge expression. | |
| Generalized | Dimension-wise score table | Reports knowledge correctness, knowledge richness, non-knowledge visual instruction following, and human MOS dimensions. |
| Slice-based heatmap | Analyzes failures by L3 branch, difficulty, knowledge operation type, education level, or knowledge quantity. |
| T2I model | Before | After | Absolute shift |
| Nano Banana 2 | 1 | 1 | 0 |
| Seedream 5.0 Lite | 2 | 3 | 1 |
| Qwen Image 2.0 | 3 | 2 | 1 |
| FLUX.2 Pro | 4 | 4 | 0 |
| Kling Image 3.0 Omni | 5 | 5 | 0 |
| Setting | Components | Case-level Agreement | VQA-level Agreement | ||||||
| Cat. | KH. | VG. | Acc. | Rec. | Prec. | Acc. | Rec. | Prec. | |
| full protocol | ✓ | ✓ | ✓ | 88.12 | 85.19 | 95.11 | 94.51 | 80.22 | 92.07 |
| w/o category | ✗ | ✓ | ✓ | 86.46 (-1.66) | 85.86 (+0.67) | 91.73 (-3.38) | 93.03 (-1.48) | 77.44 (-2.78) | 87.29 (-4.78) |
| w/o knowledge hint | ✓ | ✗ | ✓ | 86.67 (-1.45) | 86.53 (+1.34) | 91.46 (-3.65) | 92.53 (-1.98) | 77.44 (-2.78) | 84.93 (-7.14) |
| w/o visual grounding | ✓ | ✓ | ✗ | 86.07 (-2.05) | 81.82 (-3.37) | 94.92 (-0.19) | 93.56 (-0.95) | 76.17 (-4.05) | 91.07 (-1.00) |
| w/o all components | ✗ | ✗ | ✗ | 86.04 (-2.08) | 83.84 (-1.35) | 92.91 (-2.20) | 92.49 (-2.02) | 76.89 (-3.33) | 85.21 (-6.86) |
| VQA subset | Accuracy (%) |
| Required | 98.41 |
| Optional | 98.53 |
| All VQAs | 98.44 |
| Metric | Original | Rerun | Flash | GPT 5.5 |
| VQA Acc. | 94.51 | 91.74 | 92.21 | 91.33 |
| VQA Prec. | 92.07 | 83.63 | 86.74 | 86.03 |
| VQA Rec. | 80.22 | 74.44 | 73.41 | 69.11 |
| VQA | 0.8236 | 0.7366 | 0.7475 | 0.7140 |
| Case Acc. | 88.12 | 83.54 | 83.33 | 84.79 |
| Case Prec. | 95.11 | 91.29 | 92.89 | 93.41 |
| T2I model | Baseline rank | Gemini 3.1 Pro Preview (rerun) | Gemini 3.5 Flash | GPT 5.5 | |||
| Rank | Rank | Rank | |||||
| Nano Banana 2 | 1 | 1 | 1 | 1 | |||
| Seedream 5.0 Lite | 2 | 2 | 2 | 2 | |||
| Recraft V4 | 3 | 4 | 3 | 3 | |||
| HunyuanImage 3.0 | 4 | 5 | 5 | 5 | |||
| Qwen Image 2.0 | 5 | 3 | 4 | 4 | |||
| Judge setting | Kendall’s | Spearman’s | Top-3 Jaccard | Max. shift |
| Gemini 3.1 Pro Preview (rerun) | 0.8222 | 0.9273 | 0.5000 | 2 |
| Gemini 3.5 Flash | 0.8222 | 0.9394 | 1.0000 | 2 |
| GPT 5.5 | 0.8222 | 0.9394 | 1.0000 | 2 |
| Evaluation Set | Full / Eval. | Release | Coverage |
| Skill-Tree Benchmark | 1,622 / 480 | Full benchmark | All L4 atomic knowledge points |
| Generalized Benchmark | 550 / 100 | Full benchmark | All L2 subjects and L3 branches |
| General-Domain T2I Benchmark | 600 / 100 | 100-case subset only | General entities and scenes |
| Check | Source | Subset |
| Human–judge agreement | Skill-Tree 480-case subset | All 480 cases |
| Human–judge agreement | Generalized 100-case subset | 50 sampled cases |
| Human–human agreement | Skill-Tree 480-case subset | 50 sampled cases |
| Cross-judge agreement | Skill-Tree 480-case subset | All 480 cases |
| Ranking consistency | Skill-Tree unseen validation split | All 240 cases |
| Capability Axis | Representative Models |
| General-purpose T2I | Nano Banana 2 [ 35 ] ; FLUX.2 Pro [ 36 ] ; Recraft V4 [ 37 ] ; Seedream 4.5 [ 5 ] ; Hunyuan Image 3.0 [ 38 ] ; Kling Image 3.0 Omni [ 39 ] |
| Aesthetics / Visual quality | Midjourney V7 [ 40 ] ; Kling Image 3.0 Omni [ 39 ] ; Seedream 4.5 [ 5 ] |
| Text / Knowledge | Nano Banana 2 [ 35 ] ; Seedream 5.0 Lite [ 41 ] ; Wan 2.7 Pro [ 42 ] ; Qwen-Image 2.0 [ 6 , 43 ] |
| Model | Provider | Language | Output resolution |
| Qwen-Image 2.0 | Alibaba | Chinese | |
| Wan 2.7 Pro | Alibaba | Chinese | |
| FLUX.2 Pro | Black Forest Labs | English | |
| Seedream 4.5 | ByteDance | Chinese | |
| Seedream 5.0 Lite | ByteDance | Chinese | |
| Nano Banana 2 | English |
| Configurations | Dataset Construction | Automatic Evaluation | Misc | ||
| Generation | Filtering | VQA | Judge | Labeling | |
| API Entrypoint | 3.1-pro | 3-flash | 3.1-pro-customtools | 3.1-pro | 3-flash |
| Thinking Level | HIGH | UNSPECIFIED | |||
| Media Resolution | HIGH | N/A | |||
| Built-in Tools | Grounding with Google Search & URL Context | - | |||
| Custom Tools | - | - | - | - | Wikimedia Commons |
| Configuration | Gemini 3.5 Flash | GPT 5.5 |
| Model | gemini-3.5-flash | gpt-5.5 |
| Thinking / reasoning | HIGH | reasoning effort high |
| Media resolution | HIGH | not set |
| Built-in tools | Google Search and URL Context | none |
| Structured output | same as the default judge | native |
T2I Generation Evaluation
Text-to-image generation evaluation measures how accurately, safely, and consistently image models turn prompts into images.
See all 184 papers