Negation-aware image generation remains challenging for text-to-image diffusion models, which often struggle with negation due to implicit linguistic biases that are deeply entangled within text representations. As a result, bias-aligned semantics frequently dominate over intended negative constraints. To overcome this limitation, we propose a simple training-free method of Decompose and Recompose text embedding (Dero), which constructs a semantic-preserving subspace and a bias subspace using auxiliary prompts and performs semantic disentanglement based on bidirectional SVD. By decomposing entangled representations and selectively recomposing disentangled semantic components, Dero realigns text representations toward intended negation semantics while restoring residual stylistic features to preserve semantic fidelity and image quality. As a model-agnostic inference-time framework operating solely on text representations, Dero can be seamlessly applied to diverse diffusion architectures, including both U-Net and DiT based models. Extensive experiments on both object- and concept-level negation demonstrate consistent improvements in negation-aware image generation, achieving stronger text-image alignment, better semantic fidelity, and higher human preference scores.
Figures & tables
Figure 1: (a) Existing unidirectional SVD methods (e.g., SupEOT Li et al. (2024) ) suppress only bias or negation directions within padding representations in order to remove undesired information while preserving the original prompt style. However, due to the strong entanglement among semantic, stylistic, and bias components, residual biased semantics often remain in the text embedding, resulting in incomplete negation or degraded stylistic consistency. (b) In contrast, bidirectional SVD operates in a universal embedding, enabling precise removal of bias components while preserving semantic and stylistic information, thereby achieving faithful negation-aware generation.
Figure 2: Comparison of semantic disentanglement in negation. (a) Strongly coupled: negation fails, (b) Weakly coupled: style lost, (c) Uni-SVD: style degraded, and (d) Bi-SVD (Ours): negation succeeds, style preserved. For each result, the left mark indicates whether the negation constraint is satisfied, and the right mark indicates whether the original contextual style (e.g., bakery interior) is preserved.
Figure 3: Overview of DeRO. (Step 1) Biased and semantic auxiliary prompts are iteratively generated via an LLM and ranked by semantic similarity. (Step 2) Text embeddings of auxiliary prompts are encoded and decomposed via SVD to estimate the semantic basis VS(k) and bias basis VB(k) . (Step 3) Bidirectional projection constructs x^n=x^nsem+x^nstyle via two directions. Direction 1 removes bias after semantic alignment, and Direction 2 restores style after bias removal. (Step 4) The recomposed negation-aware embedding x^n is applied for image generation.
Field
Content
Prompt
a shoe without laces
Biased prompt
shoelaces
Semantic prompt
a slip-on shoe without stripes
Biased auxiliary prompts
– a sneaker with clearly tied shoelaces
– a lace-up shoe with threaded eyelets
Table 1: Examples of auxiliary prompt.
Figure 4: Comparison of object negation in different diffusion backbones. Existing methods often fail to completely remove the negated concept or degrade visual style, whereas the proposed method achieves faithful negation while preserving semantic structure and visual fidelity. Red strike-through text indicates the negated concepts.
Figure 5: Qualitative comparisons of concept-level negation in different diffusion backbones.
Figure 6: Inference-time overhead comparison in a log scale.
Figure 7: Ablation studies on Bi-SVD and semantic basis extraction. (a) Comparison between Uni-SVD and Bi-SVD on FLUX.1 and SD v1.5. (b) Performance analysis with different numbers of SVD components on FLUX.1.
Base
SVD Method
Score
TIFA
PickScore
SD1.5
Uni-SVD
0.510
20.940
Semantic Branch
0.425
20.900
Style Branch
0.100
19.504
Bi-SVD (Ours)
0.648
21.183
FLUX
Uni-SVD
0.516
21.273
Table 5: Ablation of individual branches in Bi-SVD in different diffusion backbones.
Figure 8: Comparison of different extraction methods for semantic basis decomposition. SVD provides the most stable semantic disentanglement while preserving visual quality, whereas mean aggregation shows limited separation and RPCA exhibits unstable behavior.
LLM
Prompt
Selection
TIFA
PickScore
Qwen
Noisy
Random
0.526
21.626
Top-1
0.550
21.740
Manual
Random
0.579
21.825
Top-1 (Ours)
0.650
22.088
DeepSeek
Noisy
Random
0.503
21.626
Top-1
0.612
21.750
Table 6: Robustness of auxiliary prompts across different LLMs, prompt types, and selection strategies.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of semantic entanglement of gender in profession-related prompt embeddings.
Figure 10: Comparison between Uni-SVD and Bi-SVD for negation embedding decomposition. Unlike Uni-SVD, Bi-SVD selectively preserves contextual semantics while suppressing bias-related representations.
Figure 11: LLM Instructions for Auxiliary Prompt Construction
μS=Mean(XS),μB=Mean(XB)
Appendix
Algorithm 2 Negation-aware Text Embedding via Bi-SVD
Figure 12: Qualitative comparison of Base, Uni-SVD, and Bi-SVD under varying numbers of auxiliary prompts.
Figure 13: Qualitative results of concept negation with varying numbers of auxiliary prompts.
Figure 14: Qualitative comparisons of different component decomposition methods under single and multi-negation settings.
Figure 15: Qualitative comparisons with different numbers of SVD components ( k ).
Figure 16: Qualitative comparisons for nude concept negation across SD v1.4, SD v1.5, FLUX.1, and SDXL. Regions highlighted with red markers indicate failure cases in the base model, whereas regions highlighted with blue markers correspond to improvements achieved by DeRO.
Arch
Base
Model
Latency
Time(s)
Relative
SD v1.4
U-Net
Base
2.001
-
MACE
4.389
2.194 ×
SPEED
4.534
2.266 ×
Ours
2.060
1.029 ×
SD v1.5
U-Net
Base
2.124
-
Appendix
Table 8: Comparison of inference latency based on the different diffusion backbones.
Figure 17: Example of the negation-aware user study. Participants compare generated images from three methods and evaluate negation fidelity and overall visual quality.
Figure 18: Qualitative examples on SD v1.4 on concept negation.
Figure 19: Qualitative results on SD v1.5 for object and concept negation.
Figure 20: Object negation examples on SD v3.
Figure 21: Object and artistic concept negation results on SDXL.
Figure 22: Object and concept negation results on FLUX.1-schnell.
Figure 23: Object and artistic concept negation results on FLUX.1.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Text-to-image diffusion models, which are theoretically equivalent to score-based generative models, generate images through a multi-step denoising process guided by text embeddings extracted from pretrained vision-language models such as CLIP. However, these text embeddings inherently encode social and semantic biases -- such as those related to gender and age -- that are subsequently propagated and amplified through the guidance mechanism, along with the model's training on large-scale datasets that are imbalanced with respect to these bias-related concepts, often leading to skewed outputs in text-to-image generation. We propose DebFilter, a lightweight and training-free framework for mitigating such biases in text-to-image diffusion models. Observing that the model's error prediction at each denoising step is primarily influenced by cross-attention dynamics, we introduce a bias-correction strategy that adjusts the value components within cross-attention. Specifically, we apply a fixed offset to the slice of guidance embedding, effectively steering the semantic direction of cross-attention values toward unbiased representations. This adjustment reconfigures the score landscape to produce balanced outputs while maintaining alignment with the intended text semantics. Unlike prior approaches that rely on fine-tuning or retraining, DebFilter operates entirely at inference time, requiring no additional data or model updates. Our results demonstrate that this method effectively mitigates social biases in generated images, offering an efficient and scalable pathway toward fairer and more inclusive text-to-image generation.
Seung Hyuk Lee, Songkuk Kim
School of Integrated Technology, BK21 Graduate Program in Intelligent Semiconductor Technology Yonsei University
Text-to-image (T2I) models have become increasingly capable of generating high-quality images. Yet, enforcing the explicit absence of a specified object or attribute remains a fundamentally challenging problem. Existing approaches, including prompt negation, post-hoc editing, and negative guidance, remain insufficient for explicit concept suppression, often failing to remove the target concept or degrading overall image quality. To this end, we propose Orthogonal Negative Guidance in attention feature space, a training-free method that operates in the attention output space of MM-DiT-based T2I transformers. Our method orthogonalizes negative-prompt attention features with respect to positive-prompt features and subtracts only the orthogonal component, suppressing unwanted concepts while preserving desired semantics. Experiments on FLUX-dev and FLUX-schnell show that our method achieves favorable trade-offs between concept suppression, prompt alignment, and image quality. In human evaluation, our method outperforms the second-best baseline by 18.78%. We further show that our method supports multi-concept suppression and adjustable concept suppression.
Jungmin Ko, Jungwon Park, Jimyeong Kim +3
Interdisciplinary Program in Artificial Intelligence, Seoul National University · Research Institute for Convergence Science, Seoul National University · Daegu Gyeongbuk Institute of Science and Technology +3
Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.
Dain Kim, Jinseo Kim, Sungyong Baik
Dept. of Artificial Intelligence, Hanyang University, South Korea · Dept. of Data Science, Hanyang University, South Korea