Negation without Confusion: Decomposing Bias and Recomposing Style in Text Embeddings for Negation-aware Image Generation

Sooyeon Park; Hemraj Singh; Kyunghyun Min; Youngjae Yu; Sung-Bae Cho

cs.CVOct 19, 2026

Negation without Confusion: Decomposing Bias and Recomposing Style in Text Embeddings for Negation-aware Image Generation

Authors: Sooyeon Park, Hemraj Singh, Kyunghyun Min, Youngjae Yu, Sung-Bae Cho

Organizations: ♠Yonsei University · ⋄Seoul National University

Abstract

Negation-aware image generation remains challenging for text-to-image diffusion models, which often struggle with negation due to implicit linguistic biases that are deeply entangled within text representations. As a result, bias-aligned semantics frequently dominate over intended negative constraints. To overcome this limitation, we propose a simple training-free method of Decompose and Recompose text embedding (Dero), which constructs a semantic-preserving subspace and a bias subspace using auxiliary prompts and performs semantic disentanglement based on bidirectional SVD. By decomposing entangled representations and selectively recomposing disentangled semantic components, Dero realigns text representations toward intended negation semantics while restoring residual stylistic features to preserve semantic fidelity and image quality. As a model-agnostic inference-time framework operating solely on text representations, Dero can be seamlessly applied to diverse diffusion architectures, including both U-Net and DiT based models. Extensive experiments on both object- and concept-level negation demonstrate consistent improvements in negation-aware image generation, achieving stronger text-image alignment, better semantic fidelity, and higher human preference scores.

Figures & tables

Figure 1: (a) Existing unidirectional SVD methods (e.g., SupEOT Li et al. (2024) ) suppress only bias or negation directions within padding representations in order to remove undesired information while preserving the original prompt style. However, due to the strong entanglement among semantic, stylistic, and bias components, residual biased semantics often remain in the text embedding, resulting in incomplete negation or degraded stylistic consistency. (b) In contrast, bidirectional SVD operates in a universal embedding, enabling precise removal of bias components while preserving semantic and stylistic information, thereby achieving faithful negation-aware generation.

Figure 2: Comparison of semantic disentanglement in negation. (a) Strongly coupled: negation fails, (b) Weakly coupled: style lost, (c) Uni-SVD: style degraded, and (d) Bi-SVD (Ours): negation succeeds, style preserved. For each result, the left mark indicates whether the negation constraint is satisfied, and the right mark indicates whether the original contextual style (e.g., bakery interior) is preserved.

Figure 3: Overview of DeRO. (Step 1) Biased and semantic auxiliary prompts are iteratively generated via an LLM and ranked by semantic similarity. (Step 2) Text embeddings of auxiliary prompts are encoded and decomposed via SVD to estimate the semantic basis $V^{(k)}_{\mathcal{S}}$ and bias basis $V^{(k)}_{\mathcal{B}}$ . (Step 3) Bidirectional projection constructs $\hat{x}_{n}=\hat{x}_{n}^{\mathrm{sem}}+\hat{x}_{n}^{\mathrm{style}}$ via two directions. Direction 1 removes bias after semantic alignment, and Direction 2 restores style after bias removal. (Step 4) The recomposed negation-aware embedding $\hat{x}_{n}$ is applied for image generation.

Field	Content
Prompt	a shoe without laces
Biased prompt	shoelaces
Semantic prompt	a slip-on shoe without stripes
Biased auxiliary prompts
– a sneaker with clearly tied shoelaces
– a lace-up shoe with threaded eyelets

Table 1: Examples of auxiliary prompt.

Figure 4: Comparison of object negation in different diffusion backbones. Existing methods often fail to completely remove the negated concept or degrade visual style, whereas the proposed method achieves faithful negation while preserving semantic structure and visual fidelity. Red strike-through text indicates the negated concepts.

Figure 5: Qualitative comparisons of concept-level negation in different diffusion backbones.

Figure 6: Inference-time overhead comparison in a log scale.

Figure 7: Ablation studies on Bi-SVD and semantic basis extraction. (a) Comparison between Uni-SVD and Bi-SVD on FLUX.1 and SD v1.5. (b) Performance analysis with different numbers of SVD components on FLUX.1.

Base	SVD Method	Score
		TIFA	PickScore
SD1.5	Uni-SVD	0.510	20.940
	Semantic Branch	0.425	20.900
	Style Branch	0.100	19.504
	Bi-SVD (Ours)	0.648	21.183
FLUX	Uni-SVD	0.516	21.273

Table 5: Ablation of individual branches in Bi-SVD in different diffusion backbones.

Figure 8: Comparison of different extraction methods for semantic basis decomposition. SVD provides the most stable semantic disentanglement while preserving visual quality, whereas mean aggregation shows limited separation and RPCA exhibits unstable behavior.

LLM	Prompt	Selection	TIFA	PickScore
Qwen	Noisy	Random	0.526	21.626
	Noisy	Top-1	0.550	21.740
	Manual	Random	0.579	21.825
	Manual	Top-1 (Ours)	0.650	22.088
DeepSeek	Noisy	Random	0.503	21.626
	Noisy	Top-1	0.612	21.750

Table 6: Robustness of auxiliary prompts across different LLMs, prompt types, and selection strategies.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Illustration of semantic entanglement of gender in profession-related prompt embeddings.

Figure 10: Comparison between Uni-SVD and Bi-SVD for negation embedding decomposition. Unlike Uni-SVD, Bi-SVD selectively preserves contextual semantics while suppressing bias-related representations.

A_{B}^{i}=\{a_{b}^{i,1},a_{b}^{i,2},\dots,a_{b}^{i,C}\}\sim\text{LLM}(p_{b})

Appendix

Algorithm 1 Iterative Auxiliary Prompt Selection Strategy

Panel of Figure 11 — Figure 11: LLM Instructions for Auxiliary Prompt Construction

\mu_{\mathcal{S}}=\mathrm{Mean}(X_{S}),\qquad\mu_{\mathcal{B}}=\mathrm{Mean}(X_{B})

Appendix

Algorithm 2 Negation-aware Text Embedding via Bi-SVD

Figure 12: Qualitative comparison of Base, Uni-SVD, and Bi-SVD under varying numbers of auxiliary prompts.

Figure 13: Qualitative results of concept negation with varying numbers of auxiliary prompts.

Figure 14: Qualitative comparisons of different component decomposition methods under single and multi-negation settings.

Figure 15: Qualitative comparisons with different numbers of SVD components ( $k$ ).

Figure 16: Qualitative comparisons for nude concept negation across SD v1.4, SD v1.5, FLUX.1, and SDXL. Regions highlighted with red markers indicate failure cases in the base model, whereas regions highlighted with blue markers correspond to improvements achieved by DeRO.

Arch	Base	Model	Latency
			Time(s)	Relative
SD v1.4	U-Net	Base	2.001	-
		MACE	4.389	2.194 $\times$
		SPEED	4.534	2.266 $\times$
		Ours	2.060	1.029 $\times$
SD v1.5	U-Net	Base	2.124	-

Appendix

Table 8: Comparison of inference latency based on the different diffusion backbones.

Panel of Figure 17 — Figure 17: Example of the negation-aware user study. Participants compare generated images from three methods and evaluate negation fidelity and overall visual quality.

Figure 18: Qualitative examples on SD v1.4 on concept negation.

Figure 19: Qualitative results on SD v1.5 for object and concept negation.

Figure 20: Object negation examples on SD v3.

Figure 21: Object and artistic concept negation results on SDXL.

Figure 22: Object and concept negation results on FLUX.1-schnell.

Figure 23: Object and artistic concept negation results on FLUX.1.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 290 papers

DebFilter: Eradicating Biases Stashed in Value
May 27, 2026Seung Hyuk Lee, Songkuk KimDiffusion Model Guidance Cross-Attention
School of Integrated Technology, BK21 Graduate Program in Intelligent Semiconductor Technology Yonsei University
Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation
May 28, 2026Jungmin Ko, Jungwon Park, Jimyeong Kim +3Diffusion Model Guidance T2I Generation
Interdisciplinary Program in Artificial Intelligence, Seoul National University · Research Institute for Convergence Science, Seoul National University · Daegu Gyeongbuk Institute of Science and Technology +3
Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization
Jul 1, 2026Dain Kim, Jinseo Kim, Sungyong BaikDiffusion Model Guidance Text Embeddings
Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea

May 27, 2026J/K move · Enter open · S save

DebFilter: Eradicating Biases Stashed in Value

Seung Hyuk Lee, Songkuk Kim

School of Integrated Technology, BK21 Graduate Program in Intelligent Semiconductor Technology Yonsei University

Diffusion Model Guidance Cross-Attention T2I Generation Demographic Bias in T2I Models T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models, which are theoretically equivalent to score-based generative models, generate images through a multi-step denoising process guided by text embeddings extracted from pretrained vision-language models such as CLIP. However, these text embeddings inherently encode social and semantic biases -- such as those related to gender and age -- that are subsequently propagated and amplified through the guidance mechanism, along with the model's training on large-scale datasets that are imbalanced with respect to these bias-related concepts, often leading to skewed outputs in text-to-image generation. We propose DebFilter, a lightweight and training-free framework for mitigating such biases in text-to-image diffusion models. Observing that the model's error prediction at each denoising step is primarily influenced by cross-attention dynamics, we introduce a bias-correction strategy that adjusts the value components within cross-attention. Specifically, we apply a fixed offset to the slice of guidance embedding, effectively steering the semantic direction of cross-attention values toward unbiased representations. This adjustment reconfigures the score landscape to produce balanced outputs while maintaining alignment with the intended text semantics. Unlike prior approaches that rely on fine-tuning or retraining, DebFilter operates entirely at inference time, requiring no additional data or model updates. Our results demonstrate that this method effectively mitigates social biases in generated images, offering an efficient and scalable pathway toward fairer and more inclusive text-to-image generation.

Abstract

T2I Generation

Explore similar work

DebFilter: Eradicating Biases Stashed in Value

Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation

Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization