Organizations: Harbin Institute of Technology, Shenzhen · Independent Researcher · College of Computer Science and Aritificial Intelligence, Fudan University · Institute for Artificial Intelligence, Peking University · Peng Cheng Laboratory
Recent advances in text-to-image (T2I) generation have greatly improved visual quality, yet producing images that appear visually authentic to real-world photography remains challenging. This is partly due to biases in existing evaluation paradigms: human ratings and preference-trained metrics often favor visually vivid images with exaggerated saturation and contrast, which make generations often too vivid to be real even when prompted for realistic-style images. To address this issue, we present Color Fidelity Dataset (CFD) and Color Fidelity Metric (CFM) for objective evaluation of color fidelity in realistic-style generations. CFD contains over 1.3M real and synthetic images with ordered levels of color realism, while CFM employs a multimodal encoder to learn perceptual color fidelity. In addition, we propose a training-free Color Fidelity Refinement (CFR) that adaptively modulates spatial-temporal guidance scale in generation, thereby enhancing color authenticity. Together, CFD supports CFM for assessment, whose learned attention further guides CFR to refine T2I fidelity, forming a progressive framework for assessing and improving color fidelity in realistic-style T2I generation. The dataset and code are available at https://github.com/ZhengyaoFang/CFM.
Figures & tables
Figure 1: A. Challenges in existing T2I generation and evaluation. (1) Statistical analysis shows that when prompted to produce realistic-style outputs, most T2I models generate images with higher saturation and contrast than real-world photographs. (2) A controlled saturation-scaling experiment on real photographs reveals that existing evaluation models exhibit a strong bias toward highly saturated images. The vertical axis shows the normalized score difference with respect to the reference scale (=1.0). B. Definition of color fidelity and the target of our CFM. Color fidelity measures how closely generated images preserve the natural color distribution of real-world photography, which serves as the learning objective of our proposed Color Fidelity Metric (CFM).
Figure 2: Overview of the Color Fidelity Dataset. We first apply IQA filtering process to obtain about 190K high-quality real-world images across 12 categories. Through automatic caption generation and guidance-controlled image synthesis , we further construct 1.33M images exhibiting ordered levels of color fidelity, and divide them into training and testing splits.
Figure 3: Framework of CFM and training pipeline. CFM employs Qwen2-VL as a multimodal feature backbone and an MLP projection head to map features into scalar fidelity scores. It is trained on the CFD-Training with group-wise, order-preserving samples, optimized by the softrank loss within each group.
Figure 4: Benchmark results of color fidelity evaluation across different text-to-image models. Left: category-wise CFM scores across 12 categories. Right: average CFM scores obtained by each model using our benchmark.
Model
CFD-SynPairs
CFD-Real&Syn
Traditional image quality measures
MUSIQ [ 13 ]
53.4
21.5
MANIQA [ 35 ]
54.5
24.0
CLIPIQA [ 29 ]
44.7
22.7
LIQE [ 37 ]
50.1
23.6
Generative aesthetic scoring methods
Table 1: Color fidelity prediction accuracy (%) on CFD-Test.
Figure 5: Examples of evaluation bias between existing metrics and ours. Existing metrics perform less favorably due to their training bias toward vivid, high-contrast images, whereas our proposed CFM provides a more accurate assessment of color realism, assigning higher scores to images with naturally balanced and authentic colors.
Metric
Spearman ( ρ ) ↑
Pearson ( r ) ↑
Kendall ( τ ) ↑
ImageReward [ 34 ]
62.8
63.5
49.2
PickScore [ 14 ]
71.2
72.1
56.5
MPS [ 36 ]
66.7
78.5
52.8
HPSv3 [ 18 ]
74.4
76.0
62.8
Ours
84.9
85.4
71.4
Table 2: Correlation of CFM scores on the CFD-Human.
Figure 6: Qualitative examples of different CFR variants. ‘Origin’ denotes the original image before applying CFR, while ‘CFR’ represents the results enhanced by the corresponding model-based CFR. The bottom-right value in each image indicates its CFM score, where a higher score reflects better color fidelity.
Model
Setting
FID ↓
CLIPScore ↑
Δ Sat. ↓
CFM ↑
SD3.5 [ 6 ]
−
13.3
28.2
0.15
4.9
CFR HPSv3
13.2
28.1
0.11
5.6
CFR CFM
13.1
28.2
0.07
6.9
PixArt- Σ [ 3 ]
−
16.5
27.2
0.09
4.4
CFR HPSv3
16.4
27.4
0.05
5.1
CFR CFM
16.4
27.5
0.02
6.4
Table 3: Quantitative results of CFR.
Variant
CFD-Test Acc. (%)
CFD-Human Corr.
SynPairs
Real&Syn
Spearman ↑
Kendall ↑
Pairwise loss
76.2
74.8
78.3
66.1
Visual-only
77.1
74.3
78.9
67.0
Ours
83.6
80.1
84.9
71.4
Table 4: Ablation on CFM components.
Setting
FID ↓
CLIPScore ↑
Δ Sat. ↓
CFM ↑
Baseline ( s0 )
13.3
28.2
0.15
4.9
Temporal-only
18.0
25.9
0.18
-1.3
Spatial-only
13.2
28.2
0.12
6.8
Full (ours)
13.0
28.2
0.07
6.9
Table 5: Ablation on CFR: Spatial–temporal modulation.
Figure 7: Qualitative comparison of different CFR variants on SD3.5. The temporal-only variant causes semantic inconsistency and severely distorted colors, while the spatial-only variant mitigates over-saturation but yields suboptimal detail preservation. The full spatial–temporal CFR achieves balanced and natural color rendering with the highest CFM scores.
Table 7: Category distribution of the Color Fidelity Dataset (CFD).
Figure 10: Visualization examples of CFR. The CFM score for each image is shown in the lower-right corner.
Figure 11: Visualization examples of CFD.
Figure 12: Visualization examples of CFM scores on realistic-style generations produced by different T2I models. The CFM score for each image is shown in the lower-right corner.
Figure 13: Visualization examples of CFM scores on realistic-style generations produced by different T2I models. The CFM score for each image is shown in the lower-right corner.
This paper studies the backbone used by metrics for evaluating the quality of generated images. Although generative models have advanced rapidly, some outputs still exhibit unrealistic texture characteristics, and these differences are difficult to measure. Widely used evaluation metrics provide limited insight into texture realism, since they often rely on a backbone trained for another objective, such as image classification or image-text alignment. Such objectives typically encourage invariance to low-level variations that carry texture, so the resulting features primarily encode semantic and global structure. As a result, texture-specific discrepancies may not be explicitly captured. We propose TRIG, a metric directed at the texture realism of generated images. Its backbone is instead the discriminator of a generative adversarial network pre-trained on a texture dataset, which provides a learned prior for the local statistics of natural textures. A plausibility head is fine-tuned to identify local texture degradations, while preserving natural defocus and sensor noise as positive cases. On the features of this backbone, we compute a standard distributional distance between a reference set of real images and a set of generated ones. We apply TRIG to several generative models and observe closer agreement with human realness ratings than baseline metrics.
Text-to-Image generation has evolved from basic image synthesis into a frequently used core capability in professional creative workflows, where simple text-image alignment can no longer satisfy users' pressing demands for faithful real-world reconstruction and genuine creative expression. Existing benchmarks, however, remain anchored in these foundational criteria and do not yet capture the nuanced capabilities that matter in authentic artistic practice, making it difficult to reliably distinguish state-of-the-art T2I models. To address the gap, we introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios. Qwen-Image-Bench enriches conventional evaluation with two application-driven dimensions: Real-world Fidelity and Creative Generation. Drawing on the staged reasoning inherent in professional artistic workflows, we organize these five pillars into a top-down hierarchical taxonomy that further decomposes into 23 second-level sub-capabilities and 56 third-level verifiable rubrics. To ensure broad coverage, we curate 1000 stratified prompts with each prompt jointly exercising more than four fine-grained facets across multiple pillars. We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols, that scores every image across all 56 verifiable facets, producing fine-grained, rubric-grounded, and fully attributable diagnostics rather than a single opaque score. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the two application-driven dimensions of Real-world Fidelity and Creative Generation where existing benchmarks provide little insight, while also providing a trustworthy optimization signal for production-level T2I development.
Recent advances in text-to-image (T2I) generation have led to models capable of producing highly realistic images. Yet, reliably evaluating their outputs remains challenging, especially at scale. Existing automatic evaluators, often relying on a static prompt set, struggle to capture subtle failure modes such as partial prompt misalignment, compositional errors, or visually plausible but semantically incorrect generations. In this work, we introduce DynEval, a Dynamic Evaluation framework designed to jointly assess text-to-image alignment and image quality of T2I models. To support scalable training beyond limited human-annotated data, we construct two large datasets. First, we build GenDB, a collection of 500K prompt-image pairs generated from human-written prompts drawn from DiffusionDB using a tiered prompt-model generation strategy. Second, building upon GenDB, we construct DynEvalInstruct, a 250K instruction dataset comprising prompt-image-response triplets distilled from a structured evaluation pipeline that decomposes evaluation into text-image alignment and visual quality reasoning. Using this dataset, we perform full fine-tuning of a compact evaluator through a curriculum learning strategy to effectively distill the superior evaluation capabilities of a larger teacher vision-language model, resulting in DynEval-2B and DynEval-4B. In extensive comparisons against existing evaluators across 11 benchmarks, our evaluator achieves a higher overall correlation with human judgments. Furthermore, it provides fine-grained analysis of the capabilities and failure modes of 36 T2I models across 42 subcategories and 9 semantic dimensions.
Shyam Marjit, Dheeraj Baiju, Anuj Shikarkhane +3
Indian Institute of Science · VCL, IISc · Hugging Face