cs.CVOct 23, 2026

TIE: Training-free CLIP Embedding Guidance for Compositional Text-to-Image Alignment

Authors: Sujung Hong, Tae Eun Choi, Youngjun Jun, Chanyong Yoon, Seong Jae Hwang

Organizations: Yonsei University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Text-to-image (T2I) diffusion models have advanced notably, but still fail to satisfy prompt conditions, resulting in attribute mismatches and missing objects. These errors normally fall into two categories: concept loss, when an object or attribute (i.e., concept) goes missing, and concept confusion, when an attribute is assigned to the wrong object or multiple objects blend together. Recent studies suggest that these challenges are caused by the limited capacity of CLIP text encoder to capture fine-grained semantic details. Although several methods have been proposed, concept loss and concept confusion persist in recent T2I models when handling multi-object, multi-attribute prompts. In this paper, we conduct an embedding-level analysis to develop a training-free, inference-time solution for concept loss and concept confusion across CLIP-based T2I models. We find that (1) concepts mentioned later exhibit higher embedding entropy, reflecting less concentrated representations that are more often associated with concept loss, and (2) the first CLIP attention layer captures the strength of binding between each object and its attribute. Guided by our findings, we introduce TIE, a method that improves semantic alignment through a single text-embedding update. TIE addresses concept loss via entropy-aware singular value amplification and resolves concept confusion through interpolation-extrapolation binding based on CLIP attention scores, all in a training-free manner. Extensive experiments demonstrate that TIE enhances semantic fidelity in multi-concept scenarios with minimal sampling overhead. The project page is available at https://hong-sujung.github.io/TIE/.

Figures & tables

Figure 1: Failure cases in simple prompts.
Figure 2: Analysis of positional bias and token entropy on concept loss. (a) When issuing the prompt "a ⟨attr 1 ⟩ ⟨obj 1 ⟩ and a ⟨attr 2 ⟩ ⟨obj 2 ⟩" , ⟨attr 2 ⟩ and ⟨obj 2 ⟩ is far more prone to being misaligned or missing. (b) Left: The attribute-object pair mentioned later in the prompts shows higher entropy than the pair mentioned earlier, regardless of the tokens themselves. Right: Same holds for 3-pair prompts. (c) Comparison of entropy for ⟨attr 2 ⟩ (red) and ⟨obj 2 ⟩ (blue) tokens each for when ⟨attr 2 ⟩ is Aligned, Misaligned or Missing, and when ⟨obj 2 ⟩ is Present, Mixed, or Missing, demonstrating that higher entropy is associated with concept loss.
Figure 3: Analysis of text embeddings and self-attention in the CLIP text encoder. (a) Layer-wise attribute masking shows that generation is particularly sensitive to attribute interventions at the first self-attention layer (L0), compared with later layers. (b) Conventional embedding-space metrics show substantial overlap between related and unrelated attribute–object pairs. (c) L0 attention provides a continuous signal of attribute–object relationships.
Figure 4: Overview of the proposed TIE framework. (a) Overall pipeline of TIE. (b) Adaptive Embedding Preservation (AEP), which decorrelates each concept from its preceding concepts and adaptively modifies text embeddings according to token entropy to mitigate concept loss. (c) Interpolation-Extrapolation Binding (IEB), which uses first-layer self-attention as a continuous binding signal to mitigate concept confusion.
BackboneMethodOpt- freeInference Time (s) ↓\downarrowBLIP-VQA ↑\uparrow
ColorShapeTextureSpatialComplex
SDXL  9.820.59400.50680.59610.62490.4437
 A&EXL\text{A\&E}_{XL} 62.720.59340.50820.60400.62300.4434
 SynGenXL\text{SynGen}_{XL} 22.420.59480.50590.59550.62580.4426
 ToMe 42.730.51610.35510.5621–0.2818
 Magnet✓10.660.68330.50760.59630.62390.4507
Table 1: Quantitative results of concept binding performance. All methods are evaluated with their corresponding backbones. BLIP-VQA [ 17 ] scores are measured on five categories: Color, Shape, Texture, Spatial, and Complex prompts. Inference time is compared across methods to assess efficiency. Best results are shown in bold and second-best results are underlined . For SD3.5-L, only bold formatting is used due to the limited number of compared methods.
Exp. #AEPIEBBLIP-VQA ↑\uparrow
ColorShapeTextureSpatialComplex
1  0.77660.61170.75180.68370.5050
2✓ 0.77750.61670.75290.69310.5161
3 ✓0.78310.61370.75500.69230.5171
4 (Ours)✓✓0.78260.61850.75480.69330.5154
Table 2: Ablation study on T2I-CompBench. AEP and IEB improve concept binding both individually and jointly on the SD3.5-Large backbone.
Figure 5: Qualitative comparison of TIE with SD3.5-based models. We compare TIE with other SD3.5-based models. The results show that TIE better preserves all objects, along with their spatial relationships and colors, in alignment with the given prompt.
More in this topic

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Explore similar work

CardsList
Jun 29, 2026

Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

Text-to-image (T2I) diffusion models often fail to faithfully render explicit textual descriptions, instead defaulting to strongly learned visual priors due to a phenomenon referred to as concept association bias. We show that such bias is particularly strong for one-and-only (OAO) objects, entities that exist in a single canonical form, such as celestial bodies, landmarks, and artworks. The deeply ingrained visual identity for these concepts often resists modification through prompting alone. Addressing this challenge, we first identify through an information-theoretic analysis that the final text embedding discards concept-level information present in the intermediate-layer text representations, reducing the mutual information available to the subsequent denoising process. We then propose Intermediate Text Representation (IR)-guided diffusion, which injects intermediate hidden states of the text encoder into the conditioning signal during early denoising steps, recovering suppressed concepts without any additional training, optimization, or external models. To systematically evaluate the challenging task of aligning generative outputs with unusual prompts for OAO objects, we introduce OAO-AttackBench, a benchmark comprising counterfactual prompts that directly conflict with the core visual identity of OAO objects. Experiments on four benchmarks, including OAO-AttackBench, show that our method achieves up to a 19.1 percentage-point improvement in VQAScore while preserving generation fidelity and human preference. Project page: https://soyoun-won.github.io/one-and-only-ir-guidance/.

Soyoun Won, Aryan Yazdan Parast, Basim Azam +2

The University of Melbourne, Australia

Text-Image AlignmentT2I GenerationDiffusion Guidance
AskPDF
Jun 9, 2026

STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Model

Although pretrained text-to-image (T2I) generation models can produce high-quality images, they often fail to faithfully reflect the semantic intent of complex prompts due to stochastic noise and inherent model limitations. This issue frequently manifests as the model overlooking specific objects or failing to correctly bind attributes to their corresponding entities, a challenge referred to as semantic alignment. Unlike existing approaches that rely on computationally expensive fine-tuning or labor-intensive layout priors, we propose STEDiff, a training-free method designed to enhance semantic representations directly within the text-embedding space. Specifically, we introduce a method that primarily leverages the [EOT] token to strengthen the relevant semantics of sub-sentences and then replaces the corresponding tokens in the original prompt. Furthermore, a novel semantic enhancement loss is incorporated to enforce spatial constraints, ensuring that the semantics of each entity are precisely mapped to their respective image regions. Extensive quantitative and qualitative evaluations on the T2I-CompBench demonstrate that our method notably improves semantic consistency and generation integrity in complex scenarios.

Hailan Zhang, Haipeng Liu, Bo Fu +1

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China · School of Computer and Artificial Intelligence, Liaoning Normal University, Dalian, China

Text EmbeddingsText-Image AlignmentCompositional T2I Generation
AskPDF
Jun 18, 2026

ELDiff: When Evidential Learning Meets Text-to-Image Diffusion

In multi-object text-to-image (T2I) diffusion, ensuring semantic consistency between textual prompts and generated visual content is crucial for image synthesis. However, such consistency constraint is often underemphasized in the denoising process of diffusion models. Although token supervised diffusion models can mitigate this issue by learning object-wise consistency between the image content and object segmentation maps, it tends to suffer from the problems of segmentation map bias and semantic overlap conflict, especially when involving multiple objects. In this paper, we propose ELDiff, a new evidential learning-supervised T2I diffusion model, which leverages the advantages of uncertainty metric and conflict detection to enhance the fault tolerance of unreliable segmentation maps and suppress semantic conflicts, strengthening object-wise consistency learning. Specifically, a pixel evidence loss is proposed to restrain overconfidence in unreliable labels through evidential regularization, and a token conflict loss is designed to weaken the contradiction between semantics through optimizing a measured conflict factor. Extensive experiments show that our ELDiff outperforms existing training based and train-free based T2I diffusion models on SD v1.4, SD v2.1, SDXL, SD v3.5, and Qwen-Image, without requiring additional inference-time manipulations. Notably, ELDiff can be seamlessly extended to the existing training pipeline of T2I diffusion models. Code can be found at https://github.com/QingtaoPan/ELDiff.

Qingtao Pan, Kai Ye, Zhihao Dou +2

Shandong University, Jinan, Shandong, China · Case Western Reserve University, Cleveland, OH, USA

T2I GenerationCompositional T2I GenerationT2I Diffusion Models
AskPDF