TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows

Shuai Fu; Jing Gu; Jian Zhou; Zicheng Duan; Gengze Zhou; Qi Wu

cs.CVOct 2, 2026

TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows

Authors: Shuai Fu, Jing Gu, Jian Zhou, Zicheng Duan, Gengze Zhou, Qi Wu

Organizations: AIML, Adelaide University · xAI · Responsible AI Research Centre

Abstract

Recent text-to-image models have made substantial progress in photorealism, aesthetics, and text-image alignment. Yet visually appealing images can still violate real-world plausibility, exhibiting malformed object structures, impossible anatomy, physically implausible interactions, or inconsistent spatial relationships. Such failures are not well captured by existing fidelity, aesthetics, preference, or alignment metrics. To address this gap, we introduce TerraVis, a framework for evaluating world-grounded visual consistency in generated images. TerraVis defines a structured taxonomy of world-consistency violations spanning object-, interaction-, and scene-level failures, and employs a multi-stage evaluation framework to identify and quantify them. Given an image, TerraVis first uses an MLLM to assess its eligibility for evaluation, then detects violations across 18 taxonomy-defined types and classifies them as minor or major to derive an overall world-consistency score. Across diverse open-source and proprietary text-to-image models on two widely used benchmarks, TerraVis achieves the strongest correlation with human judgments of world consistency among existing metrics. Our benchmark results further show that models that achieve strong performance on conventional metrics can still exhibit substantial world-consistency failures. These findings highlight world consistency as a complementary evaluation dimension and demonstrate that TerraVis enables systematic quantification, diagnosis, and comparison of such failures. Our code is publicly available at https://github.com/ShyFoo/TerraVis.

Figures & tables

Figure 1: World consistency is not reliably captured by existing metrics. Given the same prompt, existing metrics may assign high aesthetics, alignment, or preference scores to generated images that still contain real-world implausibilities, such as cats appearing unnaturally embedded in, or physically supported by shoes. In contrast, world consistency evaluates whether the depicted entities, structures, interactions, and spatial relationships are visually plausible with respect to the real world. The real image is taken from MSCOCO [ 27 ] and paired with the same caption used for generation.

Evaluation Dimension	Representative Methods	Evaluation Unit	World Plausibility	Detection Target
Technical quality	NIQE [ 32 ] , BRISQUE [ 31 ] , MUSIQ [ 20 ]	Single image	✗	Low-level image quality and artifacts
Aesthetics	AVA [ 33 ] , NIMA [ 44 ] , LAION-Aes [ 50 ]	Single image	✗	Visual appeal and artistic quality
Photorealism	Human rating [ 60 , 24 , 34 , 38 , 41 ]	Single image	$\LEFTcircle$	Photo-like visual appearance
Text-image alignment	CLIPScore [ 14 ] , TIFA [ 17 ] , VQAScore [ 28 ]	Text-image pair	$\LEFTcircle$	Text-image semantic consistency
Human preference	ImageReward [ 56 ] , PickScore [ 21 ] , HPS [ 55 ]	Prompt-cond. image(s)	$\LEFTcircle$	Overall subjective human preference
Distributional similarity	FID [ 15 ] , KID [ 2 ] , Precision & Recall [ 42 ]	Ref./gen. image sets	$\LEFTcircle$	Distribution-level similarity to training data

Table 1: Comparison between world consistency and common evaluation dimensions in text-to-image generation. Here,

\LEFTcircle

indicates that the dimension might partially capture world plausibility, but does not explicitly target perceptible world-consistency violations.

Figure 2: Overview of the TerraVis world-consistency violation taxonomy. TerraVis organizes world-consistency violations into three levels: object-level, interaction-level, and scene-level violations. The example images illustrate representative violation types within each level. Red ellipses indicate the regions where the corresponding violations are visually observed.

S=\exp\!\left[-\lambda\left(N_{\mathrm{maj}}+\alpha N_{\mathrm{min}}\right)\right]

Algorithm 1 TerraVis Scoring Procedure

Metric	Open-source models						Proprietary models				Pooled
	SD 3.5-Large [ 9 ]		Qwen-Image [ 52 ]		FLUX.2-dev [ 23 ]		GPT Image 1.5 [ 35 ]		Nano Banana Pro [ 46 ]		$\rho\uparrow$	$\tau\uparrow$
	$\rho\uparrow$	$\tau\uparrow$	$\rho\uparrow$	$\tau\uparrow$	$\rho\uparrow$	$\tau\uparrow$	$\rho\uparrow$	$\tau\uparrow$	$\rho\uparrow$	$\tau\uparrow$	$\rho\uparrow$	$\tau\uparrow$
Benchmark: COCO-T2I [ 57 ]
Image quality metrics
MUSIQ [ 20 ]	0.07	0.05	0.24	0.17	0.01	0.01	0.06	0.04	0.06	0.05	0.12	0.09
LAION-Aes [ 50 ]	0.18	0.13	0.08	0.06	0.09	0.06	0.30	0.23	0.14	0.12	0.21	0.15

Table 2: Correlation with human judgments of world consistency across evaluation metrics. We report Spearman’s

\rho

and Kendall’s

\tau

between automatic metrics and human ratings on two text-to-image benchmarks. Model-wise correlations are computed for each text-to-image model, while pooled correlations aggregate images from all evaluated models within each benchmark. Higher values indicate stronger agreement with human judgments. TerraVis-H is a holistic taxonomy-informed baseline that uses the full TerraVis taxonomy in a single prompt, while TerraVis decomposes the evaluation into violation identification and severity assessment. “–” indicates that the metric produced constant predictions, making rank correlations undefined.

Benchmark	Gemma 4 31B	Qwen3-VL 32B-Instruct	Gemma 4 26B-A4B	Qwen3-VL 30B-A3B-Instruct	GPT-5.5
COCO-T2I	0.42 / 0.32	0.30 / 0.25	0.37 / 0.29	0.27 / 0.23	0.44 / 0.37
GenAI-Bench	0.51 / 0.38	0.36 / 0.29	0.42 / 0.33	0.35 / 0.28	0.43 / 0.35

Table 3: Correlations with human judgments using different MLLM evaluators. Each cell reports Spearman’s

\rho

/ Kendall’s

\tau

; best results for each benchmark and metric are bolded.

Panel of Figure 3 — Figure 3: Rank heatmaps comparing T2I models across multiple evaluation metrics on two benchmarks. Human denotes the ranking derived from human judgments of world consistency. Lower ranks (with darker shades) indicate better performance.

Table 4: Ablation studies of TerraVis. (a) Per-level results are aggregated across both benchmarks. Only reports Spearman’s

\rho

with human ratings using that level alone; LOO means change in

\rho

when that level is removed. (b) Aggregation rules are evaluated using Pearson’s

r

, Spearman’s

\rho

, and Kendall’s

\tau

against human judgments. The last three rows in (b) use

\lambda=1

and

\alpha=0.5

Panel of Figure 4 — Figure 4: Four patterns of disagreement in world-consistency evaluation. (a) TerraVis detects genuine violations overlooked by all three human raters. (b) TerraVis produces likely false-positive detections in an image that all three human raters judge fully consistent. (c) TerraVis fails to detect anatomical distortions in the children’s hands and arms that human raters identify and our manual review confirms. (d) Human raters assign substantially different scores to the same image. The violation descriptions in (a) and (b) are reproduced verbatim from TerraVis outputs.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Violation type	Question inserted into {QUESTION}
Object identity	Ignoring embedded depictions such as murals, paintings, screens, or printed images, does the image contain any physically present entity that is presented as a real object or living being but has impossible identity traits, such as a living dragon or an impossible animal/object hybrid?
Structural distortion	Does any object appear visibly malformed due to generation failure, for example melted, broken in shape, or geometrically distorted in an implausible way?
Biological anatomy	Does any human or animal body show anatomically impossible features, such as an impossible limb orientation, or eyes looking in incompatible directions?
Non-biological structure	Does any non-living object have an impossible structure or part arrangement, such as disconnected wheels, misplaced parts, physically invalid construction, or parts whose geometry or alignment is inconsistent under the apparent viewpoint?
Texture/surface	Does any object show a physically implausible material or surface pattern, such as skin that looks like wood or stone without scene justification?
Symbolic content	Does any visible text, sign, symbol, logo, or marking appear broken, distorted, or unrecognizable in a way that violates real-world plausibility?

Appendix

Table A1 : Violation-specific questions used to instantiate the prompt template in Section B.2 .

Figure A1 : AMT interface for rating world-grounded visual consistency. Annotators first read the descriptions of taxonomy-guided criteria covering object-level, interaction-level, and scene-level violations, and then answer whether the image contains a world-consistency violation and assign a score from 1 to 5. "N/A" instruction is also added in the rating scale section.

Benchmark	Evaluator	Positive rate (%)	Precision (%) $\uparrow$	Recall (%) $\uparrow$
COCO-T2I	Human majority vote	48.6	–	–
	TerraVis-H	8.7	86.0	15.4
	TerraVis	33.3	75.8	51.9
GenAI-Bench	Human majority vote	69.7	–	–
	TerraVis-H	14.8	93.3	19.8
	TerraVis	37.8	89.1	48.3

Appendix

Table A1 : Positive rates and world-consistency violation detection performance on COCO-T2I and GenAI-Bench. Precision and recall are computed against human majority-vote labels. The highest precision and recall within each benchmark are bolded.

Violation type	Only : $\rho$	LOO : $\Delta$
Object (all 7)	0.384	−0.1488
Object identity	0.275	−0.0330
Structural distortion	0.241	−0.0419
Biological anatomy	0.109	−0.0012
Non-biological structure	0.124	−0.0044
Texture/surface	0.068	−0.0026

Appendix

Table A2 : Ablation of individual violation types in TerraVis, with results aggregated across COCO-T2I and GenAI-Bench. Only reports Spearman’s

\rho

with human ratings using each type alone; LOO reports the change in

\rho

when that type is removed. Underline highlights the individual violation type with the largest marginal contribution to correlation with human ratings.

Setting	$\lambda$	$\alpha$	Pearson’s $r$	Spearman’s $\rho$	Kendall’s $\tau$
Best Pearson	0.9	0.5	0.4230	0.4281	0.3471
Best Spearman	–	0.3	0.3826	0.4284	0.3478
Best Kendall	–	1.0	0.3840	0.4264	0.3481
Default	1.0	0.5	0.4229	0.4281	0.3477
Range over grid	–	–	0.3797–0.4230	0.4234–0.4284	0.3463–0.3481

Appendix

Table A3 : Sensitivity analysis of TerraVis aggregation hyperparameters. Rank correlations remain stable across a broad range of

\lambda

and

\alpha

. The default setting

(\lambda=1,\alpha=0.5)

achieves near-optimal performance.

Evaluator pair	COCO-T2I ( $\tau$ )	GenAI-Bench ( $\tau$ )
Group 1
GPT-5.5 – Gemma 4 31B	0.80	0.80
Group 2
Gemma 4 26B-A4B – Qwen3-VL-30B-A3B-Instruct	1.00	0.80
Gemma 4 26B-A4B – Qwen3-VL-32B-Instruct	0.80	0.80
Qwen3-VL-30B-A3B-Instruct – Qwen3-VL-32B-Instruct	0.80	1.00

Appendix

Table A4 : Pairwise Kendall’s

\tau

between T2I model rankings produced by TerraVis using different MLLM evaluators. Group 1 comprises GPT-5.5 and Gemma 4 31B, while Group 2 comprises the remaining three evaluators. Shaded rows highlight the agreement within Group 1 and the mean pairwise agreement within Group 2 .

Metric	Judge / backbone	Throughput (images/s) $\uparrow$	Average time (s/image) $\downarrow$	Relative time $\downarrow$
RAHF	ViT-L + T5 multi-head	43.06	0.023	$1.0\times$
HPSv3	Qwen2-VL 7B	15.35	0.065	$2.8\times$
VQAScore	Gemma 4 31B	13.35	0.075	$3.2\times$
TerraVis (ours)	Gemma 4 31B	0.855	1.17	$50.4\times$
TIFA	Gemma 4 31B	0.799	1.25	$53.9\times$

Appendix

Table A5 : Computational efficiency on 200 randomly sampled images using a single NVIDIA H100 GPU. Average time per image is the reciprocal of throughput; relative time is normalized to RAHF, the fastest method. Higher throughput and lower processing time indicate better efficiency.

Figure A1 : Share of detected violations by level. For each model, bars give the percentage of its TerraVis-detected violations that fall at the object, interaction, and scene levels on COCO-T2I (left) and GenAI-Bench (right); shares sum to 100% for each model and benchmark. Object-level violations dominate for every model, yet the interaction and scene levels together still account for 27.5–50.0% of all violations.

Figure A2 : Share of detected violations by type. The shares in Figure A1 broken down into the 18 violation types. The top seven rows are object-level types, the next eight interaction-level, and the bottom three scene-level; shares sum to 100% for each model and benchmark. The breakdown exposes model-specific profiles that recur on both benchmarks, e.g., structural distortion for Qwen-Image and object-context and relative scale for GPT Image 1.5.

Panel of Figure 19 — Figure A3 : Four patterns of disagreement in world-consistency evaluation. (a) TerraVis detects genuine violations overlooked by human raters. (b) TerraVis incorrectly detects violations in images that all human raters judge fully consistent. (c) TerraVis fails to detect violations that human raters identify and our manual review confirms. (d) Human raters assign substantially different scores to the same image. The violation descriptions in (a) and (b) are reproduced verbatim from TerraVis outputs.

Panel of Figure 20 — Figure A4 : Prompt alignment versus world consistency. TIFA and VQAScore assess prompt alignment, which does not necessarily reflect world consistency. (a) Images receive high alignment scores despite low human world-consistency ratings. (b) Images that deviate from their prompts receive low alignment scores despite high human world-consistency ratings. These examples highlight the distinction between prompt alignment and world consistency. For example, the third image in (a) matches the requested juxtaposition of ancient buildings and futuristic transport, yet receives low world-consistency scores, illustrating that prompt faithfulness does not guarantee world plausibility.

Panel of Figure 21 — Figure A5 : HPSv3 versus world consistency. (a) Images receive high preference scores despite low human world-consistency ratings. (b) Images receive low preference scores despite high human world-consistency ratings. HPSv3 is shown on its original scale; TerraVis and normalized human scores use $[0,1]$ . These examples highlight the distinction between human preference and world consistency. For instance, the third example in (a), depicting an ancient library with floating books and “Secrets of the Ages” inscribed on the archway, receives a high HPSv3 preference score despite low world-consistency scores from both TerraVis and human raters.

Panel of Figure 22 — Figure A6 : RAHF versus world consistency. RAHF’s overall score and its plausibility, alignment, and aesthetics scores are reported separately. (a) Images receive high RAHF scores despite low human world-consistency ratings. (b) Images receive low RAHF scores despite high human ratings. RAHF’s plausibility head was trained to detect artifacts using RichHF-18K [ 26 ] , which limits its coverage to the types of artifacts represented in the training set and may not generalize well to images generated by newer text-to-image models. For example, the first image in (b) suggests that RAHF incorrectly penalizes an artifact-free and world-consistent image generated by Nano Banana Pro.

Explore similar work

May 2, 2025

WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation

Recent text-to-image (T2I) models achieve high visual fidelity and accurate explicit grounding from input prompts. However, image generation is fundamentally a process of semantic expansion. Beyond surface-level descriptions, generating authentic visual content requires supplementing prompts with unstated cues, a crucial step that current models frequently overlook. In response, we introduce WorldGenBench, a benchmark designed to evaluate how well T2I systems integrate factual grounding with contextual inference. Unlike prior work that isolates these abilities, WorldGenBench emphasizes their synergy, recognizing that true understanding requires both knowing and reasoning. To enable interpretable evaluation, we propose the Knowledge Checklist Score, which assesses whether generated images faithfully reflect key explicit and implicit semantic elements. Evaluating 24 models reveals persistent gaps: proprietary models like GPT-4o show strong integration, while most open-source systems lag, particularly in culturally diverse and scientific scenarios. Our analysis further identifies common failure modes, including semantic hallucination and the underrepresentation of non-Western knowledge, underscoring the limitations of current models in achieving real-world readiness. These findings highlight the need to guide future T2I models toward deeper, knowledge-aware generation. Project Page: https://dwanzhang-ai.github.io/WorldGenBench/

Ruoshi Xu, Daoan Zhang, Che Jiang +7

Southern University of Science and Technology · University of Rochester · New York University +2

T2I Generation T2I Generation Evaluation

Ask PDF

Mar 29, 2026

ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks

Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce ImagenWorld, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.

Samin Mahdizadeh Sani, Max Ku, Nima Jamali +23

University of Waterloo · G-G-G · Independent +1

Image Generation Evaluation T2I Generation Evaluation Image Editing Evaluation

Ask PDF

May 1, 2025

Multimodal Language Models as Text-to-Image Model Evaluators

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.

Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5

1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3

LLM-as-a-Judge Benchmark Design Automated Evaluation Vision-Language Model-as-a-Judge T2I Generation Evaluation

Ask PDF