Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.
Figures & tables
Figure 1: When the target concept lies in a low-density region, the generated samples tend to drift toward semantically dominant, high-density concepts [ 33 ] in the learned score space, resulting in the suppression of rare or compositional attributes. Our proposed adaptive coefficient γt∗ , derived from auxiliary prompt blending between c~T and its anchor c~A , dynamically corrects this bias and produces target-faithful results. Unlike a fixed coefficient γt , the adaptive γt∗ adjusts per timestep to maintain a target-aligned denoising trajectory.
Figure 2: Toy Example of the target concept generation. (a) Training distributions: frequent samples N((0,−6),I) (orange) and rare-prior samples N((0,3),1.5I) (purple) with mean positions marked by crosses; (b) Generated samples using fixed linear interpolation plerp(x∣c~A,c~T;γt=0.8) ; (c) Generated samples using adaptive interpolation p(x∣c~A,c~T;γt∗) ; (d) 2-Wasserstein distance between generated distributions and the target N((0,3),1.5I) as a function of interpolation parameter γt (blue line). The distance curve shows that while fixed interpolation achieves a minimum distance around γt≈0.8 , the adaptive method (red dashed line) consistently outperforms any fixed interpolation choice.
Single Object
Multi Objects
Models
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
SD1.5 [ 26 ]
55.0
38.8
33.8
23.1
36.9
23.1
24.4
36.3
33.9
SDXL [ 22 ]
60.0
56.9
71.3
47.5
58.1
39.4
35.0
47.5
52.0
PixArt [ 5 ]
49.4
58.8
76.9
56.3
63.1
35.6
30.0
48.1
52.3
SD3.0 [ 6 ]
49.4
76.3
53.1
71.9
65.0
55.0
51.2
70.0
61.5
FLUX [ 2 ]
58.1
71.9
47.5
52.5
60.0
55.0
48.1
70.3
57.9
Table 1: Text-to-image alignment performances in the RareBench with other baselines with GPT-4o based evaluation. Best values are denoted with bold , second-best with underlined .
Figure 3: Qualitative comparison with state-of-the-art diffusion models on RareBench. All models are executed with the same random seed. Our method achieves stronger text-to-image alignment without additional training.
Figure 4: Qualitative comparison of image editing results using FlowEdit [ 15 ] and our method. All edits are performed with the same random seed. Compared to FlowEdit, our approach better preserves source content while faithfully applying the instructed edits.
CLIP-T ↑
CLIP-I ↑
LPIPS ↓
DINO ↑
DreamSim ↓
SDEdit 0.2 [ 19 ]
0.330
0.885
0.251
0.634
0.213
SDEdit 0.4
0.340
0.854
0.316
0.564
0.273
ODE Inv [ 27 ]
0.337
0.813
0.318
0.549
0.326
iRFDS [ 37 ]
0.335
0.822
0.376
0.534
0.327
FlowEdit [ 15 ]
0.344
0.872
0.181
0.719
0.259
Ours
0.341
0.905
0.155
0.814
0.155
Table 2: Quantitative results on FlowEdit dataset. CLIP-T ↑ evaluates text adherence, while CLIP-I ↑ , LPIPS ↓ , DINO ↑ , and DreamSim ↓ assess structure preservation. Our method achieves superior performance on structure preservation while maintaining strong text alignment.
Figure 5: Comparison between fixed γt , R2F [ 21 ] , and our adaptive coefficient on RareBench. The blue line denotes fixed γt , the green line denotes R2F, and the red line represents our adaptive approach, which consistently outperforms the fixed baseline.
Figure 6: Trade-off between structure preservation (CLIP-I) and text alignment (CLIP-T) across different fixed γt values in FlowEdit dataset. Our method consistently resides in a Pareto-favorable region, balancing structural preservation and text alignment more effectively than fixed baselines.
Models
Property
Shape
Texture
Action
Complex
Avg
SD3.0
49.4
76.3
53.1
71.9
65.0
63.1
Human Generated (R2F)
79.8
68.8
76.3
78.5
-
75.9
Human Generated (Ours)
93.1
74.4
83.7
79.2
-
82.6
Arbitrary (R2F)
57.5
80.0
63.1
70.0
69.4
68.0
Arbitrary (Ours)
65.6
83.7
62.5
73.1
70.0
71.0
“objects” (R2F)
90.0
78.1
80.6
78.8
80.6
81.6
Table 3: Experiments with different anchor prompt strategies. Our method consistently outperforms R2F [ 21 ] regardless of the anchor prompts.
Single Object
Multi Objects
Models
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
SDXL
SDXL [ 22 ]
60.0
56.9
71.3
47.5
58.1
39.4
35.0
47.5
52.0
LMD [ 17 ]
23.8
35.6
27.5
23.8
35.6
33.1
34.4
33.1
30.9
RPG [ 36 ]
33.8
54.4
66.3
31.9
37.5
21.9
15.6
29.4
36.4
ELLA [ 11 ]
31.3
61.6
64.4
43.1
66.3
42.5
50.6
51.9
51.5
R2F (SDXL) [ 21 ]
71.3
77.5
73.8
54.4
70.6
50.6
36.0
52.8
60.9
Table 4: Text-to-image alignment on RareBench with SDXL, SD3.0, and IterComp, comparing our method integrated into each model. Results demonstrate consistent robustness across diverse pre-trained diffusion backbones.
Figure 7: Qualitative comparison on different pre-trained diffusion baselines, SDXL, SD3.0, and IterComp. Comparing our method integrated with these pre-trained models.
Figure 8: Quantitative comparison on varying classifier-free guidance scale w on the FlowEdit dataset.
Figure 9: Evolution of the adaptive coefficient γt∗ across diffusion timesteps in image editing (FlowEdit). The coefficient exhibits clear saturation in later steps as structural guidance from the source image becomes dominant.
Figure 10: Evolution of the adaptive coefficient γt∗ across diffusion timesteps in rare concept generation (RareBench). Unlike image editing, γt∗ maintains stable values throughout the denoising process, reflecting continuous need for anchor stabilization.
Strategy
∥d∥∥
∥d⊥∥
Total ↓
T2I ↑
Arbitrary
42.2
47.9
90.1
71.0
Objects
32.5
26.6
59.1
83.1
Human Generated
30.2
28.4
58.6
82.6
GPT-4o
27.3
28.0
55.3
87.9
LLaMA3
26.8
27.0
53.8
81.0
Table 5: Anchor quality comparison across different anchor generation strategies. Lower displacement indicates anchors closer to the target with fewer artifacts. T2I scores are evaluated using GPT-4o. All correlations are negative.
Models
RareBench Multi
Concat
Relation
Complex
SD3.0
55.0
51.2
70.0
R2F+
74.4
63.7
64.8
Ours (R2F+)
83.8
83.6
80.3
Table 6: Quantitative comparison on RareBench-Multi between R2F+ and our method built upon the R2F+ baseline.
Models
Rare Concept
Image Editing
SD3
R2F
Ours
FlowEdit
Ours
Peak Memory (GB)
31.52
31.76
32.22
19.97
19.97
GPU Time (sec)
27.14
26.12
37.94
7.41
7.54
Table 7: GPU time and memory required to generate an image.
Figure 11: Comparison of prompt reconstruction between R2F and our method. R2F often collapses multiple rare concepts into a single entity, leading to entangled generations. In contrast, our method explicitly preserves each rare concept by directly pairing it with its frequent counterpart, resulting in disentangled and faithful generations.
Single Object
Multi Objects
Models
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
SD3 [ 6 ]
46.3
58.3
46.0
43.5
53.1
40.8
46.7
60.5
49.4
R2F (SD3) [ 21 ]
71.9
66.2
71.4
63.7
69.3
42.6
44.6
67.3
62.1
Ours (SD3)
76.7
68.5
71.9
65.0
70.1
59.8
49.7
68.7
66.3
Table 9: Qualitative comparison based on our user study. For each prompt, participants were shown multiple images generated by different models in random order and without model names (Fig. 14 ). Participants selected the preferred result according to the evaluation criteria define in Tab. 8 .
Figure 12: Qualitative results for face-accessory image editing. We extract the source prompt using GPT and augment the target prompt by adding “glasses” as an accessory. Our method preserves facial details more faithfully than FlowEdit.
Models
LAION-aesthetic
ImageReward
PickScore
SD3.0
6.317±0.190
0.915±0.151
21.771±0.462
R2F
6.215±0.220
1.048±0.266
22.155±0.239
Ours
6.191±0.166
1.123±0.328
22.177±0.222
Table 10: Aesthetic and preference evaluation on RareBench . LAION-aesthetic measures visual appeal, ImageReward models human preference, and PickScore evaluates text–image alignment. Our method achieves the best preference and alignment scores.
Figure 13: Failure results on RareBench where both ours and R2F exhibit attribute-object mismatches. This behaviors aligns with previously reported limitations in CLIP’s ability to bind compositional concpets faithfully [ 16 ] .
< System Prompt >
You are a helper language model for a text-to-image generation program that aims to create images based on input text. The program often struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts.
< User Prompt >
Extract rare concepts from the input text and replace them with relevant yet more frequent ones. Perform the following process step by step:
a. Identify and extract any rare concepts from the provided input text. If the text contains one or more rare concepts, extract them all. If there are no rare concepts present, do not extract any concepts. The extracted rare concepts should not overlap.
Table 11: Full LLM instruction for AAPB to generate target-to-anchor concept mappings for rare concept generation.
Input: A banana shaped apple
Output:
Num Rare Concepts: 1
a. Rare concept: A banana shaped apple
b. A banana shaped apple does not exist in reality, while the possibility of a banana shaped red object existing is much higher. Main noun object: apple, Replaced frequent object: red object
c. A banana shaped red object BREAK A banana shaped apple
Final Prompt Sequence: A banana shaped red object BREAK A banana shaped apple
Table 12: In-context examples of the LLM prompt for AAPB.
Figure 14: User study interface for evaluating text-to-image alignment. Participants were asked to rate how well each generated image matched the given text prompt based on semantic accuracy and visual consistency. The image order was randomized, and model identities were hidden to avoid bias.
Figure 15: Ablation study with varying classifier-free guidance scale w .
Figure 16: Uncurated qualitative visualizations of our method on RareBench, generated from eight randomly selected prompts across all categories and eight random seeds.
Figure 17: Qualitative results of the image editing.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.
Text-to-image (T2I) diffusion models have achieved striking progress but still struggle to synthesize rare concepts involving unusual attribute-object pairings, often resulting in concept omission or semantic drift where a dominant entity overwhelms the generation. Tracing these failures to a lack of compositional balance during the denoising trajectory, we propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process. RADIANCE augments pretrained backbones with three modular components: (1) a Compositional Similarity Monitor (CSM) that tracks the emergence of objects and attributes in intermediate latents via CLIP-based feedback; (2) a Bidirectional Scale Controller (BSC) that applies a reactive "restoring force" using positive and negative IP-Adapter scales to rebalance biased trajectories; and (3) a Feedback Guidance Scheduler (FGS) that coordinates these updates across timesteps without additional training. We further extend the framework to multi-object prompts via Delayed Adapter Activation (DAA) and Layer-wise Alternating Guidance (LAG) to prevent premature concept fusion. By overlapping monitoring and denoising through pipelined execution, RADIANCE maintains competitive latency while significantly enhancing the per-sample success rate and effective throughput. Experiments on RareBench and T2I-CompBench demonstrate that RADIANCE consistently enhances compositional alignment and perceptual quality over state-of-the-art baselines.
Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance
Nikolai Röhrich, Isabell Hans, Felix Krause +1
LMU Munich, Germany · Konrad Zuse School of Excellence in Reliable AI (relAI), Germany · Munich Center for Machine Learning (MCML), Germany