Guiding Diffusion Models with Semantically Degraded Conditions

Shilong Han; Yuming Zhang; Hongxia Wang

cs.CVMar 11, 2026

Guiding Diffusion Models with Semantically Degraded Conditions

Authors: Shilong Han, Yuming Zhang, Hongxia Wang

Organizations: College of Science, National University of Defense Technology

Abstract

Classifier-Free Guidance (CFG) is a cornerstone of modern text-to-image models, yet its reliance on a semantically vacuous null prompt ( $\varnothing$ ) generates a guidance signal prone to geometric entanglement. This is a key factor limiting its precision, leading to well-documented failures in complex compositional tasks. We propose Condition-Degradation Guidance (CDG), a novel paradigm that replaces the null prompt with a strategically degraded condition, $\boldsymbol{c}_{\text{deg}}$ . This reframes guidance from a coarse "good vs. null" contrast to a more refined "good vs. almost good" discrimination, thereby compelling the model to capture fine-grained semantic distinctions. We find that tokens in transformer text encoders split into two functional roles: content tokens encoding object semantics, and context-aggregating tokens capturing global context. By selectively degrading only the former, CDG constructs $\boldsymbol{c}_{\text{deg}}$ without external models or training. Validated across diverse architectures including Stable Diffusion 3, FLUX, and Qwen-Image, CDG markedly improves compositional accuracy and text-image alignment. As a lightweight, plug-and-play module, it achieves this with negligible computational overhead. Our work challenges the reliance on static, information-sparse negative samples and establishes a new principle for diffusion guidance: the construction of adaptive, semantically-aware negative samples is critical to achieving precise semantic control. Code is available at https://github.com/Ming-321/Classifier-Degradation-Guidance.

Figures & tables

Figure 1: Qualitative comparison between Classifier-Free Guidance (CFG) and our Condition-Degradation Guidance (CDG) across three state-of-the-art models (SD3, SD3.5, and Flux). These examples demonstrate CDG’s superior capability in handling complex compositional prompts where CFG often fails. CDG consistently outperforms CFG in accurate text rendering, precise spatial relationships and attribute binding, as well as complex object interactions.

Figure 2: CDG synthesizes a geometrically superior guidance signal compared to CFG. (Top) Geometric Decoupling: CDG maintains near-perfect orthogonality throughout generation, while CFG suffers from significant early-stage entanglement. (Bottom) Interference Energy Ratio: CDG exhibits minimal interference, in stark contrast to CFG’s substantial energy waste in misaligned directions. Together, these analyses demonstrate that CDG’s guidance signal is structurally cleaner and more efficient from its inception, explaining its enhanced compositional control.

Figure 3: An illustration of our proposed pipeline for constructing the semantically degraded condition, $\boldsymbol{c}_{\text{deg}}$ . The process begins with attention graph extraction (a–c), where the self-attention map (b) from a transformer block (a) is modeled as a graph (c). Next, the Weighted PageRank (WPR) algorithm is applied to compute an importance score for each token (d). Following our Stratified Degradation strategy, these scores are used to generate a binary mask $\boldsymbol{m}$ (e). Finally, the mask facilitates the construction of $\boldsymbol{c}_{\text{deg}}$ via masked interpolation (f) between the original condition $\boldsymbol{c}$ and the null condition $\emptyset$ .

Figure 4: WPR reveals a clear importance dichotomy between content and context-aggregating tokens: content tokens carry fine-grained semantics while context-aggregating tokens carry coarse-grained semantics, as exemplified by the prompt “A man is cooking, MineCraft Style.” (a) The stem plot shows that high importance scores (red for CLIP, cyan for T5) are almost exclusively concentrated on semantic content tokens. (b) The ranked list confirms that the top tokens (“minecraft”, “cooking”, “man”) are almost all content-related. This dichotomy motivates our Stratified Degradation strategy, which first degrades content tokens and then context-aggregating tokens for controllable semantic degradation.

Method		FID $\downarrow$	CLIP Score $\uparrow$	Aesthetic Score $\uparrow$	VQA Score $\uparrow$
SD3	CFG	35.69	31.73	5.66	91.44
	CADS	36.16	31.72	5.65	91.44
	ICG	39.09	31.41	5.79	90.89
	SEG	41.90	30.28	5.56	84.15
	PAG	50.60	30.15	5.52	81.27
	SFG	38.92	31.72	5.52	90.52

Table 1: Quantitative comparison on the MS-COCO 2017 validation set. Results are shown for SD3, SD3.5, FLUX.1, and Qwen-Image. Best results in bold , second-best underlined . (

\downarrow

) indicates lower is better, (

\uparrow

) higher is better.

Method		Spatial $\uparrow$	Comp $\uparrow$	Differ $\uparrow$	Univ $\uparrow$
SD3	CFG	78.66	74.08	77.22	70.74
	CADS	78.55	74.05	76.98	70.11
	ICG	78.09	73.10	75.89	69.39
	SEG	72.26	69.51	70.46	65.52
	PAG	69.65	66.40	67.09	64.80
	CDG	79.84	74.86	78.26	71.53

Table 2: GenAI-Bench compositional reasoning results: Spatial Relation (Spatial), Comparison (Comp), Differentiation (Differ), and Universal (Univ).

Components		Metrics
Importance	Stratified	FID $\downarrow$	CLIP Score $\uparrow$	Aesthetic Score $\uparrow$	VQA Score $\uparrow$
✓	✓	33.89	31.98	5.68	92.21
✗	✓	34.17	32.02	5.68	92.27
✓	✗	35.06	30.93	5.48	86.31
✓ ∗	✗	50.73	29.86	5.22	80.10
✗	✗	47.02	30.39	5.21	83.55

Table 3: Ablation study on CDG core components at fixed degradation budget (

R_{\text{deg}}=1.1

). ‘✓’ and ‘✗’ denote active and inactive components. Asterisk ( ∗ ) indicates reverse ranking (least important tokens).

Figure 5: Hyperparameter analysis: joint effect of intervention block (

\lambda_{\text{block}}

) and Degradation Ratio (

R_{\text{deg}}

) on SD3.

Figure 6: (a) Semantic degradation sequence under CFG* as $R_{\text{deg}}$ increases from 0.00 to 2.00. (b) Quantitative analysis.

Method	Time (s)	Aesthetic	VQA
CFG (Baseline)	5.456	5.66	91.44
CDG (Per-Step)	8.031 (+47.2%)	5.68	92.33
CDG	5.655 (+3.6%)	5.68	92.21

Table 4: Efficiency comparison of CDG implementation variants (one-time vs. per-step WPR computation) on SD3 with

R_{\text{deg}}=1.1

\lambda_{\text{block}}=1

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Token importance scores computed via cross-attention summation for the prompt “A man is cooking, Minecraft Style.” The method incorrectly assigns highest importance to padding tokens rather than semantic content tokens.

Model	Approach	$w$	$T$	$s$	seed	$\sigma$	$\lambda_{\text{block}}$	$r_{\text{content}}\|r_{\text{CtxAgg}}$
SD3	CFG	7	28	-	-	-	-	-
	CADS	7	28	0.07	-	-	-	-
	ICG	7	28	-	42	-	-	-
	PAG	3	28	-	-	-	13	-
	SEG	3	28	-	-	5	13	-
	CDG	7	28	-	-	-	1	1.0 $\|$ 0.0

Appendix

Table 5: Optimal hyperparameters for different models. (“-” denotes not used)

Figure 8: Hyperparameter ablation on SD3.5 and FLUX.1: Performance metrics (CLIP Score and VQA Score) as a function of Degradation Ratio

R_{\text{deg}}

. Top row: CLIP Score; bottom row: VQA Score. Left column: SD3.5 results; right column: FLUX.1 results. For SD3.5, both metrics achieve their best performance around

R_{\text{deg}}=1.0

1.3

, exhibiting a stable plateau that is consistent with the SD3 analysis in the main text. For FLUX.1, metrics exhibit relative stability across the tested range, with

R_{\text{deg}}=1.0

providing a reliable initialization point.

Figure 9: Visual comparison of results generated by varying the degradation ratio $R_{\text{deg}}$ .

Figure 10: Comparison between unconditional generation (left) and context-aggregating-token-only guidance (right) for four prompts. The right column uses CFG ∗ with $\boldsymbol{c}_{\text{deg}}$ constructed by $r_{\text{content}}=1,r_{\text{CtxAgg}}=0$ .

Model	Method	Counting	Comparison	Differentiation	Negation	Universal	Avg
SD3	CFG	76.00	74.08	77.22	46.58	70.74	68.92
	CADS	75.59	74.05	76.98	46.74	70.11	68.70
	ICG	75.37	73.10	75.89	47.46	69.39	68.24
	SEG	70.87	69.43	70.87	47.90	64.96	64.81
	PAG	67.76	66.40	67.09	47.84	64.80	62.78
	CDG (Ours)	76.50	74.86	78.26	46.24	71.53	69.48

Appendix

Table 6: Comprehensive GenAI-Bench evaluation results for advanced reasoning skills. For each skill dimension within each model, the best method is shown in bold and the second-best is underlined. CDG achieves the best performance in most dimensions, demonstrating its superior reasoning capabilities.

Model	Method	Action Relation	Attribute	Scene	Spatial Relation	Part Relation	Avg
SD3	CFG	79.06	77.23	76.67	78.66	76.35	77.59
	CADS	78.84	77.05	76.51	78.55	76.36	77.46
	ICG	77.95	76.42	75.77	78.09	74.96	76.64
	SEG	72.57	71.98	71.28	73.50	70.99	72.06
	PAG	68.58	68.35	66.95	69.65	68.75	68.46
	CDG (Ours)	79.29	77.56	77.18	79.84	76.19	78.01

Appendix

Table 7: Comprehensive GenAI-Bench evaluation results for basic compositional skills. For each skill dimension within each model, the best method is shown in bold and the second-best is underlined. CDG consistently outperforms baseline methods across different models and skill types.

Method	FID $\downarrow$	CLIP $\uparrow$	Aes $\uparrow$	VQA $\uparrow$
CFG	35.69	31.73	5.66	91.44
CDG-C (CLIP only)	32.68	32.01	5.65	92.32
CDG-T (T5 only)	34.81	31.94	5.67	92.11
CDG (Both)	34.05	32.00	5.70	92.40

Appendix

Table 8: Encoder-specific ablation on SD3. -C/-T: CLIP/T5 only degradation.

$R_{\text{deg}}$	FID $\downarrow$		CLIP $\uparrow$		Aes $\uparrow$		VQA $\uparrow$
$R_{\text{deg}}$	WPR	Rand ∗	WPR	Rand ∗	WPR	Rand ∗	WPR	Rand ∗
1.1	33.80	34.17	31.98	32.02 $\pm$ 0.52	5.68	5.68 $\pm$ 0.16	92.27	92.27 $\pm$ 0.88
0.9	35.08	33.73	31.72	31.90 $\pm$ 0.95	5.67	5.67 $\pm$ 0.28	91.05	91.65 $\pm$ 1.57

Appendix

Table 9: WPR vs. Random ranking ablation on SD3.

Figure 11: Synergy with PAG: CDG provides the core semantic foundation, while PAG refines local details.

Figure 12: CDG for Text-Guided Editing: Correcting semantic errors where the baseline fails.

Figure 13: CDG for Controllable Generation: Providing semantic precision to render complex details under structural constraints.

Figure 14: More visual results on the SD 3 model.

Figure 15: More visual results on the SD 3.5 model.

Figure 16: More visual results on the Flux model.

Explore similar work

DICE: Distilling Classifier-Free Guidance into Text Embeddings
Feb 6, 2025Zhenyu Zhou, Defang Chen, Can Wang +2Diffusion Model Alignment Diffusion Model Guidance
Zhejiang University, State Key Laboratory of Blockchain and Data Security · University at Buffalo, State University of New York
ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts
Nov 26, 2024Jinho Chang, Changsun Lee, Hyungjin Chung +1Diffusion Model Guidance Classifier-Free Guidance
Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea · EverEx, Seoul, South Korea
Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
Aug 17, 2026Artem Sergievskii, Artyom Turevich, Sergey KastryulinDiffusion Model Guidance Classifier-Free Guidance
HSE University · Yandex, HSE University

Feb 6, 2025J/K move · Enter open · S save

DICE: Distilling Classifier-Free Guidance into Text Embeddings

Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen+1

Zhejiang University, State Key Laboratory of Blockchain and Data Security · University at Buffalo, State University of New York

Diffusion Model Alignment Diffusion Model Guidance Diffusion Model Distillation Diffusion Model Inference Acceleration T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models are capable of generating high-quality images, but suboptimal pre-trained text representations often result in these images failing to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, CFG introduces significant computational overhead. In this paper, we present DIstilling CFG by sharpening text Embeddings (DICE) that replaces CFG in the sampling process with half the computational complexity while maintaining similar generation quality. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Furthermore, examining the enhancement pattern, we identify the underlying mechanism of DICE that sharpens specific components of text embeddings to preserve semantic information while enhancing fine-grained details. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL, and PixArt-

α

demonstrate the effectiveness of our method. Code is available at https://github.com/zju-pi/dice.

Abstract

Explore similar work

DICE: Distilling Classifier-Free Guidance into Text Embeddings

ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models