Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation

Dian Xie; Shitong Shao; Lichen Bai; Zikai Zhou; Bojun Cheng; Shuo Yang; Jun Wu; Zeke Xie

cs.CVFeb 26, 2026

Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation

Authors: Dian Xie, Shitong Shao, Lichen Bai, Zikai Zhou, Bojun Cheng, Shuo Yang, Jun Wu, Zeke Xie

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology (Shenzhen) · Cogniser Information Technology

Abstract

Classifier-free guidance (CFG) has helped diffusion models achieve great conditional generation in various fields. Recently, more diffusion guidance methods have emerged with improved generation quality and human preference. However, can these emerging diffusion guidance methods really achieve solid and significant improvements? In this paper, we rethink recent progress on diffusion guidance. Our work mainly consists of four contributions. First, we reveal a critical evaluation pitfall that common human preference models exhibit a strong bias towards large guidance scales. Simply increasing the CFG scale can easily improve quantitative evaluation scores due to strong semantic alignment, even if image quality is severely damaged (e.g., oversaturation and artifacts). Second, we introduce a novel guidance-aware evaluation (GA-Eval) framework that employs effective guidance scale calibration to enable fair comparison between current guidance methods and CFG by identifying the effects orthogonal and parallel to CFG effects. Third, motivated by the evaluation pitfall, we design Transcendent Diffusion Guidance (TDG) method that can significantly improve human preference scores in the conventional evaluation framework but actually does not work in practice. Fourth, in extensive experiments, we empirically evaluate recent eight diffusion guidance methods within the conventional evaluation framework and the proposed GA-Eval framework. Notably, simply increasing the CFG scales can compete with most studied diffusion guidance methods, while all methods suffer severely from winning rate degradation over standard CFG. Our work would strongly motivate the community to rethink the evaluation paradigm and future directions of this field.

Figures & tables

Panel of Figure 1 — Figure 1: The HPS v2 scores of generated images under different CFG scales $\omega\in\{5.5,10,15,20\}$ , respectively. Model: Stable Diffusion-XL. HPSv2 exhibit a strong bias to large CFG scales in a wide range, even if generation quality starts to degrade due to too strong guidance.

Figure 2: Visual comparison of different methods with their corresponding results under effective guidance scale $\omega^{\mathrm{e}}$ . The e-CFG method, namely standard CFG with effective CFG scales calibrated by GA-Eval, can easily achieve performance improvement comparable to most recent diffusion guidance or sampling methods.

Figure 3: Different evaluation metrics under different guidance scales. Model: Stable Diffusion-XL. Dataset: Pick-a-Pic. Except AES and PickScore, other metrics would give higher ratings on images generated with a larger guidance scale within

\omega\in[5.5,20]

Figure 4: Comparison of classifier-free guidance Ho and Salimans (2021) and transcendent diffusion guidance (TDG).

Figure 5: The winning rate

\eta^{\text{e-CFG}}

compared to effective CFG and their degradation

\Delta\eta

of different methods on HPD dataset. Left:

\eta^{\text{e-CFG}}

. Right:

\Delta\eta

. Among all methods, applying

\omega^{\mathrm{e}}

has minor influence to APG. Meanwhile, AES demonstrated negative

\Delta\eta

on many methods, which means AES would give worse ratings to images generated with a large guidance scale.

Method	HPS v2	ImageReward	AES	PickScore	CLIPScore	Average $\eta$	$\omega^{\mathrm{e}}$
Pick-a-Pic	( $\eta^{\text{CFG}}/\eta^{\text{e-CFG}}/\Delta\eta$ )					( $\eta^{\text{CFG}}$ / $\eta^{\text{e-CFG}}$ )
Z-Sampling	90% / 74% / 16%	75% / 63% / 12%	65% / 76% /-11%	72% / 73% / -1%	64% / 59% / 5%	73% / 69%	13.51
CFG++	82% / 68% / 14%	70% / 58% / 12%	39% / 53% /-14%	56% / 57% / -1%	60% / 54% / 6%	61% / 58%	8.91
SAG	75% / 59% / 16%	45% / 34% / 11%	64% / 66% / -2%	62% / 56% / 6%	53% / 44% / 9%	60% / 52%	8.14
TDG	62% / 48% / 14%	61% / 46% / 15%	49% / 42% / 7%	51% / 51% / 0%	63% / 53% / 10%	57% / 48%	8.27
SEG	60% / 56% / 4%	59% / 49% / 10%	57% / 56% / 1%	35% / 33% / 2%	47% / 42% / 5%	52% / 47%	6.10

Table 1: The winning rates

\eta^{\text{CFG}},\eta^{\text{e-CFG}}

and its degradation

\Delta\eta

of different methods on various datasets. Model: Stable Diffusion-XL (

\omega=5.5

). We sort these methods based on average

\eta^{\text{e-CFG}}

, labeling those greater than 55% in blue and the others in red . Most of methods have

\omega^{\mathrm{e}}>\omega

, and suffering from severe winning rate degradation. Only recent Z-Sampling can still maintain high winning rates

\eta^{\text{e-CFG}}

after setting CFG scale as

\omega^{\mathrm{e}}

, while some methods even cannot compete with effective CFG.

Method	Single object ( $\uparrow$ )	Two object ( $\uparrow$ )	Counting ( $\uparrow$ )	Colors ( $\uparrow$ )	Position ( $\uparrow$ )	Color attribution ( $\uparrow$ )	Overall ( $\uparrow$ )	$\omega^{\mathrm{e}}$
CFG	98.91%	75.63%	37.66%	85.11%	10.87%	26.88%	55.84%	5.5
TDG	100.00%	76.77%	53.75%	87.23%	11.00%	21.00%	58.29%	8.28
e-CFG	100.00%	73.74%	47.50%	86.17%	9.00%	27.00%	57.23%	8.28
Z-Sampling	100.00%	75.76%	46.25%	84.04%	17.00%	20.00%	57.18%	16.05
e-CFG	98.75%	77.78%	36.25%	85.11%	16.00%	25.00%	56.48%	16.05
CFG++	100.00%	81.82%	46.25%	85.11%	16.00%	20.00%	58.20%	8.81

Table 2: The quantitative results of different methods on GenEval. Model: Stable Diffusion-XL (

\omega=5.5

). We labeled methods surpassed effective CFG with blue , and others with red . Increasing the CFG scale can also significantly increase the GenEval scores.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The qualitative results of TDG.

Figure 7: Comparison of HPS v2 Score of TDG and CFG at different timesteps. Model: SD-3.5.

Figure 8: TDG vs. CFG in manifold space. Compared to CFG, TDG enlarges the search space to a hyperplane.

Figure 9: TDG hyperparameter grid search heatmap, with HPSv2 winning rate compared to CFG as the criteria. Model: SD-XL.

Figure 10: Different mask ratio of prompt tokens in TDG. Model: SD-XL.

Panel of Figure 11 — Figure 11: The HPS v2 scores of generated images under different CFG scales $\omega\in\{5.5,10,15,20\}$ , respectively. Model: Stable Diffusion-XL. HPSv2 exhibit a strong bias to large CFG scales in a wide range, even if generation quality starts to degrade due to too strong guidance.

Method	HPS v2	ImageReward	AES	PickScore	CLIPScore	Average $\eta$	$\omega^{\mathrm{e}}$
Pick-a-Pic	( $\eta^{\text{CFG}}/\eta^{\text{e-CFG}}/\Delta\eta$ )					( $\eta^{\text{CFG}}$ / $\eta^{\text{e-CFG}}$ )
PAG	53% / 53% / 0%	44% / 43% / 1%	53% / 57% / -4%	36% / 43% / -7%	44% / 41% / 3%	46% / 47%	5.33
TDG	61% / 48% / 13%	50% / 51% / -1%	45% / 51% / -6%	38% / 47% / -9%	53% / 47% / 6%	49% / 49%	6.62
Z-Sampling	40% / 36% / 4%	41% / 54% /-13%	46% / 49% / -3%	31% / 37% / -6%	53% / 58% / -5%	42% / 47%	9.57
DrawBench	( $\eta^{\text{CFG}}/\eta^{\text{e-CFG}}/\Delta\eta$ )					( $\eta^{\text{CFG}}$ / $\eta^{\text{e-CFG}}$ )
PAG	51% / 50% / 2%	44% / 40% / 3%	61% / 64% / -3%	42% / 40% / 2%	39% / 42% / -3%	48% / 47%	5.41

Appendix

Table 3: The winning rate and its degradation of different methods on various datasets. Model: Stable Diffusion-3.5 (

\omega=4.5

Method	HPS v2	ImageReward	AES	PickScore	CLIPScore	Average $\eta$	$\omega^{\mathrm{e}}$
Pick-a-Pic	( $\eta^{\text{CFG}}/\eta^{\text{e-CFG}}/\Delta\eta$ )					( $\eta^{\text{CFG}}$ / $\eta^{\text{e-CFG}}$ )
Z-Sampling	90% / 60% / 30%	80% / 62% / 18%	70% / 63% / 7%	84% / 73% / 11%	71% / 46% / 25%	79% / 61%	11.80
CFG++	82% / 62% / 20%	73% / 58% / 15%	58% / 55% / 3%	65% / 49% / 16%	57% / 55% / 2%	67% / 56%	9.09
FreeU	64% / 61% / 3%	73% / 67% / 6%	63% / 57% / 6%	52% / 45% / 7%	50% / 47% / 3%	60% / 55%	6.57
PAG	48% / 45% / 3%	49% / 41% / 8%	54% / 55% / -1%	49% / 45% / 4%	46% / 44% / 2%	49% / 46%	5.62
APG	11% / 31% /-20%	21% / 43% /-22%	49% / 53% / -4%	26% / 34% / -8%	45% / 58% /-13%	30% / 44%	11.19

Appendix

Table 4: The winning rates and the degradation of different methods on various datasets. Model: Stable Diffusion-2.1 (

\omega=5.5

Dataset	Method	HPS v2 ( $\uparrow$ )	ImageReward ( $\uparrow$ )	AES ( $\uparrow$ )	PickScore ( $\uparrow$ )	CLIPScore ( $\uparrow$ )
Pick-a-Pic	CFG	28.51	0.845	6.112	22.50	31.01
	APG	28.70	0.937	5.995	22.53	32.28
	e-CFG	29.30	1.032	5.992	22.61	32.23
	CFG++	29.21	1.025	6.091	22.60	31.46
	e-CFG	29.05	0.990	6.103	22.64	31.39
	FreeU	28.88	0.929	6.156	22.11	30.01

Appendix

Table 5: Quantitative results of different methods. Model: Stable Diffusion-XL (

\omega=5.5

Dataset	Method	HPS v2 ( $\uparrow$ )	ImageReward ( $\uparrow$ )	AES ( $\uparrow$ )	PickScore ( $\uparrow$ )	CLIPScore ( $\uparrow$ )
Pick-a-Pic	CFG	26.25	-0.353	5.658	20.14	27.08
	APG	26.03	-0.441	5.675	20.05	31.97
	e-CFG	26.00	-0.420	5.537	20.17	30.29
	CFG++	27.13	0.045	5.751	20.43	27.72
	e-CFG	26.99	-0.030	5.731	20.45	27.49
	FreeU	26.93	-0.004	5.759	20.34	27.08

Appendix

Table 6: Quantitative results of different methods. Model: Stable Diffusion-2.1 (

\omega=5.5

Dataset	Method	HPS v2 ( $\uparrow$ )	ImageReward ( $\uparrow$ )	AES ( $\uparrow$ )	PickScore ( $\uparrow$ )	CLIPScore ( $\uparrow$ )
Pick-a-Pic	CFG	28.43	1.004	5.921	21.94	28.72
	PAG	28.49	0.918	5.994	21.77	28.19
	e-CFG	28.54	1.053	5.925	21.96	28.92
	TDG	28.56	0.951	5.890	21.82	28.57
	e-CFG	28.56	1.016	5.921	21.87	28.83
	Z-Sampling	28.30	0.980	5.853	21.63	28.74

Appendix

Table 7: Quantitative results of different methods. Model: Stable Diffusion-3.5 (

\omega=4.5

Method	FID ( $\downarrow$ )	HPS v2 ( $\uparrow$ )	AES ( $\uparrow$ )	$\omega^{\mathrm{e}}$
CFG	13.56	28.73	5.558	5.5
Z-Sampling	17.32	29.56	5.666	10.81
e-CFG	14.21	29.22	5.550	10.81
CFG++	17.27	29.46	5.643	8.99
e-CFG	16.07	29.28	5.609	8.99

Appendix

Table 8: Quantitative results of different methods on COCO-30K. Model: Stable Diffusion-XL (

\omega=5.5

Method	FID ( $\downarrow$ )	IS ( $\uparrow$ )	AES ( $\uparrow$ )	$\omega^{\mathrm{e}}$
CFG	21.22	474.9	4.911	4
Z-Sampling	25.43	471.7	4.853	9.17
e-CFG	24.04	465.1	4.834	9.17
CFG++	24.33	480.1	4.950	8.01
e-CFG	25.55	480.0	4.791	8.01

Appendix

Table 9: Quantitative results of different methods on ImageNet-50K. Model: DiT-XL/2 (

\omega=4

Method	HPS v3	OneIG
Pick-a-Pic	( $\eta^{\text{CFG}}/\eta^{\text{e-CFG}}/\Delta\eta$ )
PAG	71% / 68% / 3%	44% / 44% / 0%
CFG++	55% / 54% / 1%	36% / 26% / 10%
Z-Sampling	79% / 67% / 12%	54% / 55% / -1%
APG	44% / 39% / 5%	24% / 26% / -2%
FreeU	39% / 35% / 4%	26% / 21% / 5%

Appendix

Table 10: The winning rate and its degradation of different methods on Pick-a-Pic datasets. Model: Stable Diffusion-XL (

\omega=5.5

Dataset	Model	HPS v3 ( $\uparrow$ )	UnifiedReward (to CFG)	UnifiedReward (to e-CFG)
Pick-a-Pic	CFG	6.837	/	/
	PAG	7.781	0.16	0.16
	e-CFG	7.341	/	/
	CFG++	7.648	0.1	-0.02
	e-CFG	7.612	/	/
	Z-Sampling	7.776	0.38	0.4

Appendix

Table 11: Quantitative results of different methods on Pick-a-Pic dataset. Model: Stable Diffusion-XL (

\omega=5.5

Guidance Scale	Alignment ( $\uparrow$ )	Diversity ( $\uparrow$ )	Reasoning ( $\uparrow$ )	Style ( $\uparrow$ )	Text ( $\uparrow$ )
5.5	.5638	.4399	.1932	.3325	.0128
6.0	.5738	.4357	.1910	.3328	.0178
6.5	.5771	.4294	.1970	.3374	.0170
7.0	.5820	.4242	.2000	.3420	.0164
7.5	.5837	.4225	.2076	.3363	.0181
8.0	.5936	.4169	.2048	.3366	.0166

Appendix

Table 12: Quantitative results of SD-XL on OneIG-Bench with different guidance scales.

Figure 12: Reward evaluation on images with different saturation, each color represents images with different saturation. 20 images were randomly picked in MS-COCO dataset with changed saturation. Model: SD-XL.

Figure 13: Reward evaluation on images with different saturation, each color represents images generated by a prompt. 20 prompts were randomly selected in Pick-a-Pic, DrawBench, and HPD, and corresponding images were generated with different guidance scale. Model: SD-XL.

Panel of Figure 14 — Figure 14: Effective guidance scale value over timestep. Model: SD-XL. Dataset: Pick-a-Pic. Note that except Z-Sampling, CFG++, and APG, which modified the noise scheduling process, other methods use EulerDiscreteScheduler by default.

Figure 15: Qualitative Results of Z-Sampling.

Figure 16: Qualitative Results of CFG++.

Figure 20: Qualitative Results of FreeU.

Explore similar work

Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
Aug 17, 2026Artem Sergievskii, Artyom Turevich, Sergey KastryulinDiffusion Model Guidance Classifier-Free Guidance
HSE University · Yandex, HSE University
Guiding Diffusion Models with Semantically Degraded Conditions
Mar 11, 2026Shilong Han, Yuming Zhang, Hongxia WangDiffusion Model Guidance T2I Diffusion Models
College of Science, National University of Defense Technology
Adversarial Learning of Classifier-Free Guidance Schedules
Aug 14, 2026Ashwini Pokle, Alexandre Galashov, Arnaud Doucet +2Classifier-Free Guidance T2I Diffusion Models
Google DeepMind

Aug 17, 2026J/K move · Enter open · S save

Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models

Artem Sergievskii, Artyom Turevich, Sergey Kastryulin

HSE University · Yandex, HSE University

Diffusion Model Guidance Classifier-Free Guidance Text-Image Alignment Attention Control in Diffusion Models

Open paper AskPDF arXiv

Inference-time quality-enhancement methods are an effective and widely adopted means of improving diffusion models without expensive retraining. We study a family of training-free techniques conceptually rooted in Classifier-Free Guidance (CFG), most of which were originally proposed on older U-Net diffusion models and validated using metrics that assess image quality in isolation, without accounting for compositional alignment or semantic correspondence between the generated image and its associated text prompt. We re-evaluate eight such methods on two open-weight rectified-flow transformers under a fixed per-model protocol and three compositional-alignment benchmarks. No method consistently improves on CFG across the measured criteria. APG obtains several nominal best scores, but the corresponding gains often remain within the estimated evaluation uncertainty. Attention-perturbation methods provide isolated gains on SD3.5 Medium and more frequent degradations on FLUX.2 [klein] 4B Base, while CFG remains a competitive lower-cost baseline.

Abstract

Explore similar work

Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models

Guiding Diffusion Models with Semantically Degraded Conditions

Adversarial Learning of Classifier-Free Guidance Schedules