cs.CVMar 19, 2026
SaveAdaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
Organizations: Hanyang University, South Korea
Abstract
Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.
Figures & tables
Figure 1: When the target concept lies in a low-density region, the generated samples tend to drift toward semantically dominant, high-density concepts [ 33 ] in the learned score space, resulting in the suppression of rare or compositional attributes. Our proposed adaptive coefficient , derived from auxiliary prompt blending between and its anchor , dynamically corrects this bias and produces target-faithful results. Unlike a fixed coefficient , the adaptive adjusts per timestep to maintain a target-aligned denoising trajectory.
Figure 2: Toy Example of the target concept generation. (a) Training distributions: frequent samples (orange) and rare-prior samples (purple) with mean positions marked by crosses; (b) Generated samples using fixed linear interpolation ; (c) Generated samples using adaptive interpolation ; (d) 2-Wasserstein distance between generated distributions and the target as a function of interpolation parameter (blue line). The distance curve shows that while fixed interpolation achieves a minimum distance around , the adaptive method (red dashed line) consistently outperforms any fixed interpolation choice.
| Single Object | Multi Objects | ||||||||
| Models | Property | Shape | Texture | Action | Complex | Concat | Relation | Complex | Avg |
| SD1.5 [ 26 ] | 55.0 | 38.8 | 33.8 | 23.1 | 36.9 | 23.1 | 24.4 | 36.3 | 33.9 |
| SDXL [ 22 ] | 60.0 | 56.9 | 71.3 | 47.5 | 58.1 | 39.4 | 35.0 | 47.5 | 52.0 |
| PixArt [ 5 ] | 49.4 | 58.8 | 76.9 | 56.3 | 63.1 | 35.6 | 30.0 | 48.1 | 52.3 |
| SD3.0 [ 6 ] | 49.4 | 76.3 | 53.1 | 71.9 | 65.0 | 55.0 | 51.2 | 70.0 | 61.5 |
| FLUX [ 2 ] | 58.1 | 71.9 | 47.5 | 52.5 | 60.0 | 55.0 | 48.1 | 70.3 | 57.9 |
Table 1: Text-to-image alignment performances in the RareBench with other baselines with GPT-4o based evaluation. Best values are denoted with bold , second-best with underlined .
Figure 3: Qualitative comparison with state-of-the-art diffusion models on RareBench. All models are executed with the same random seed. Our method achieves stronger text-to-image alignment without additional training.
Figure 4: Qualitative comparison of image editing results using FlowEdit [ 15 ] and our method. All edits are performed with the same random seed. Compared to FlowEdit, our approach better preserves source content while faithfully applying the instructed edits.
| CLIP-T | CLIP-I | LPIPS | DINO | DreamSim | |
|---|---|---|---|---|---|
| SDEdit 0.2 [ 19 ] | 0.330 | 0.885 | 0.251 | 0.634 | 0.213 |
| SDEdit 0.4 | 0.340 | 0.854 | 0.316 | 0.564 | 0.273 |
| ODE Inv [ 27 ] | 0.337 | 0.813 | 0.318 | 0.549 | 0.326 |
| iRFDS [ 37 ] | 0.335 | 0.822 | 0.376 | 0.534 | 0.327 |
| FlowEdit [ 15 ] | 0.344 | 0.872 | 0.181 | 0.719 | 0.259 |
| Ours | 0.341 | 0.905 | 0.155 | 0.814 | 0.155 |
Table 2: Quantitative results on FlowEdit dataset. CLIP-T evaluates text adherence, while CLIP-I , LPIPS , DINO , and DreamSim assess structure preservation. Our method achieves superior performance on structure preservation while maintaining strong text alignment.
Figure 5: Comparison between fixed , R2F [ 21 ] , and our adaptive coefficient on RareBench. The blue line denotes fixed , the green line denotes R2F, and the red line represents our adaptive approach, which consistently outperforms the fixed baseline.
Figure 6: Trade-off between structure preservation (CLIP-I) and text alignment (CLIP-T) across different fixed values in FlowEdit dataset. Our method consistently resides in a Pareto-favorable region, balancing structural preservation and text alignment more effectively than fixed baselines.
| Models | Property | Shape | Texture | Action | Complex | Avg |
|---|---|---|---|---|---|---|
| SD3.0 | 49.4 | 76.3 | 53.1 | 71.9 | 65.0 | 63.1 |
| Human Generated (R2F) | 79.8 | 68.8 | 76.3 | 78.5 | - | 75.9 |
| Human Generated (Ours) | 93.1 | 74.4 | 83.7 | 79.2 | - | 82.6 |
| Arbitrary (R2F) | 57.5 | 80.0 | 63.1 | 70.0 | 69.4 | 68.0 |
| Arbitrary (Ours) | 65.6 | 83.7 | 62.5 | 73.1 | 70.0 | 71.0 |
| “objects” (R2F) | 90.0 | 78.1 | 80.6 | 78.8 | 80.6 | 81.6 |
Table 3: Experiments with different anchor prompt strategies. Our method consistently outperforms R2F [ 21 ] regardless of the anchor prompts.
| Single Object | Multi Objects | |||||||||
| Models | Property | Shape | Texture | Action | Complex | Concat | Relation | Complex | Avg | |
| SDXL | SDXL [ 22 ] | 60.0 | 56.9 | 71.3 | 47.5 | 58.1 | 39.4 | 35.0 | 47.5 | 52.0 |
| LMD [ 17 ] | 23.8 | 35.6 | 27.5 | 23.8 | 35.6 | 33.1 | 34.4 | 33.1 | 30.9 | |
| RPG [ 36 ] | 33.8 | 54.4 | 66.3 | 31.9 | 37.5 | 21.9 | 15.6 | 29.4 | 36.4 | |
| ELLA [ 11 ] | 31.3 | 61.6 | 64.4 | 43.1 | 66.3 | 42.5 | 50.6 | 51.9 | 51.5 | |
| R2F (SDXL) [ 21 ] | 71.3 | 77.5 | 73.8 | 54.4 | 70.6 | 50.6 | 36.0 | 52.8 | 60.9 | |
Table 4: Text-to-image alignment on RareBench with SDXL, SD3.0, and IterComp, comparing our method integrated into each model. Results demonstrate consistent robustness across diverse pre-trained diffusion backbones.
Figure 7: Qualitative comparison on different pre-trained diffusion baselines, SDXL, SD3.0, and IterComp. Comparing our method integrated with these pre-trained models.
Figure 8: Quantitative comparison on varying classifier-free guidance scale on the FlowEdit dataset.
Figure 9: Evolution of the adaptive coefficient across diffusion timesteps in image editing (FlowEdit). The coefficient exhibits clear saturation in later steps as structural guidance from the source image becomes dominant.
Figure 10: Evolution of the adaptive coefficient across diffusion timesteps in rare concept generation (RareBench). Unlike image editing, maintains stable values throughout the denoising process, reflecting continuous need for anchor stabilization.
| Strategy | Total | T2I | ||
|---|---|---|---|---|
| Arbitrary | 42.2 | 47.9 | 90.1 | 71.0 |
| Objects | 32.5 | 26.6 | 59.1 | 83.1 |
| Human Generated | 30.2 | 28.4 | 58.6 | 82.6 |
| GPT-4o | 27.3 | 28.0 | 55.3 | 87.9 |
| LLaMA3 | 26.8 | 27.0 | 53.8 | 81.0 |
Table 5: Anchor quality comparison across different anchor generation strategies. Lower displacement indicates anchors closer to the target with fewer artifacts. T2I scores are evaluated using GPT-4o. All correlations are negative.
| Models | RareBench Multi | ||
|---|---|---|---|
| Concat | Relation | Complex | |
| SD3.0 | 55.0 | 51.2 | 70.0 |
| R2F+ | 74.4 | 63.7 | 64.8 |
| Ours (R2F+) | 83.8 | 83.6 | 80.3 |
Table 6: Quantitative comparison on RareBench-Multi between R2F+ and our method built upon the R2F+ baseline.
| Models | Rare Concept | Image Editing | |||
|---|---|---|---|---|---|
| SD3 | R2F | Ours | FlowEdit | Ours | |
| Peak Memory (GB) | 31.52 | 31.76 | 32.22 | 19.97 | 19.97 |
| GPU Time (sec) | 27.14 | 26.12 | 37.94 | 7.41 | 7.54 |
Table 7: GPU time and memory required to generate an image.
Figure 11: Comparison of prompt reconstruction between R2F and our method. R2F often collapses multiple rare concepts into a single entity, leading to entangled generations. In contrast, our method explicitly preserves each rare concept by directly pairing it with its frequent counterpart, resulting in disentangled and faithful generations.
| Single Object | Multi Objects | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Models | Property | Shape | Texture | Action | Complex | Concat | Relation | Complex | Avg |
| SD3 [ 6 ] | 46.3 | 58.3 | 46.0 | 43.5 | 53.1 | 40.8 | 46.7 | 60.5 | 49.4 |
| R2F (SD3) [ 21 ] | 71.9 | 66.2 | 71.4 | 63.7 | 69.3 | 42.6 | 44.6 | 67.3 | 62.1 |
| Ours (SD3) | 76.7 | 68.5 | 71.9 | 65.0 | 70.1 | 59.8 | 49.7 | 68.7 | 66.3 |
Table 9: Qualitative comparison based on our user study. For each prompt, participants were shown multiple images generated by different models in random order and without model names (Fig. 14 ). Participants selected the preferred result according to the evaluation criteria define in Tab. 8 .
Figure 12: Qualitative results for face-accessory image editing. We extract the source prompt using GPT and augment the target prompt by adding “glasses” as an accessory. Our method preserves facial details more faithfully than FlowEdit.
| Models | LAION-aesthetic | ImageReward | PickScore |
|---|---|---|---|
| SD3.0 | 6.317±0.190 | 0.915±0.151 | 21.771±0.462 |
| R2F | 6.215±0.220 | 1.048±0.266 | 22.155±0.239 |
| Ours | 6.191±0.166 | 1.123±0.328 | 22.177±0.222 |
Table 10: Aesthetic and preference evaluation on RareBench . LAION-aesthetic measures visual appeal, ImageReward models human preference, and PickScore evaluates text–image alignment. Our method achieves the best preference and alignment scores.
Figure 13: Failure results on RareBench where both ours and R2F exhibit attribute-object mismatches. This behaviors aligns with previously reported limitations in CLIP’s ability to bind compositional concpets faithfully [ 16 ] .
| System Prompt |
| You are a helper language model for a text-to-image generation program that aims to create images based on input text. The program often struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts. |
| User Prompt |
| Extract rare concepts from the input text and replace them with relevant yet more frequent ones. Perform the following process step by step: |
| a. Identify and extract any rare concepts from the provided input text. If the text contains one or more rare concepts, extract them all. If there are no rare concepts present, do not extract any concepts. The extracted rare concepts should not overlap. |
Table 11: Full LLM instruction for AAPB to generate target-to-anchor concept mappings for rare concept generation.
| Input: A banana shaped apple |
| Output: |
| Num Rare Concepts: 1 |
| a. Rare concept: A banana shaped apple |
| b. A banana shaped apple does not exist in reality, while the possibility of a banana shaped red object existing is much higher. Main noun object: apple, Replaced frequent object: red object |
| c. A banana shaped red object BREAK A banana shaped apple |
| Final Prompt Sequence: A banana shaped red object BREAK A banana shaped apple |
Table 12: In-context examples of the LLM prompt for AAPB.
Figure 14: User study interface for evaluating text-to-image alignment. Participants were asked to rate how well each generated image matched the given text prompt based on semantic accuracy and visual consistency. The image order was randomized, and model identities were hidden to avoid bias.
Figure 15: Ablation study with varying classifier-free guidance scale .
Figure 16: Uncurated qualitative visualizations of our method on RareBench, generated from eight randomly selected prompts across all categories and eight random seeds.
Figure 17: Qualitative results of the image editing.
More in this topic
T2I Generation
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
See all 299 papers