Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.
Figures & tables
Figure 2: Overview of our ADAPT framework. (a) Our framework introduces three complementary zero-shot control modules in the Stable Diffusion 3 (SD3) architecture. Adaptive Prompt Scheduling (APS) determines optimal stop points Si based on each token’s spatial attention map score SAttn at each step; Pooled Embedding Manipulation (PEM) operates on the CLIP’s pooled text embedding prior to modulation in the SD3 pipeline; Latent Space Manipulation (LSM) is applied at the feature level directly within the transformer block after attention computation. (b) The overview of attention scoring. We compute Top-k attention response scores s(k) to determine when each token’s semantic content has been sufficiently established. (c) Visualization of adaptive prompt scheduling between yprog and target prompt ytar based on stop points Si .
Single Object
Multi Objects
Models
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
SD1.5 [ 20 ]
55.0
38.8
33.8
23.1
36.9
23.1
24.4
36.3
33.9
SDXL [ 16 ]
60.0
56.9
71.3
47.5
58.1
39.4
35.0
47.5
52.0
PixArt [ 4 ]
49.4
58.8
76.9
56.3
63.1
35.6
30.0
48.1
52.3
SD3.0 [ 6 ]
49.4
76.3
53.1
71.9
65.0
55.0
51.2
70.0
61.5
FLUX [ 1 ]
58.1
71.9
47.5
52.5
60.0
55.0
48.1
70.3
57.9
Table 1: Text-to-image alignment performances in the RareBench with other baselines with GPT-4o based evaluation. Best values are notated with bold , second-best with underlined .
Single
Multi
Method
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
R2F (SD3)
89.4
79.4
81.9
80.0
72.5
70.0
58.8
73.8
75.7
PEM (w/o Adaptive)
90.0
84.4
78.8
71.9
76.9
71.9
73.8
79.4
78.4
PEM
92.5
91.3
78.8
69.4
78.1
70.6
77.5
80.0
79.8
PEM + LSM
92.5
91.3
80.6
71.9
77.5
73.1
76.9
79.4
80.4
PEM + APS
96.3
88.8
80.0
77.5
78.1
74.4
71.3
79.4
80.7
Table 2: Ablation studies demonstrating the incremental contribution of each proposed component (PEM, LSM, and APS) to text-to-image alignment performance across different evaluation categories on the RareBench dataset.
Figure 3: Qualitative comparison between previous methods and our method across rare concept prompts from RareBench. Our approach significantly improves upon R2F in a zero-shot setting, demonstrating superior alignment between image and text for a wide range of challenging attributes and compositions. All samples are generated with the same random seed (42) for fair comparison.
Single
Multi
Method
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
Simple ytar
49.4
76.3
53.1
71.9
65.0
55.0
51.2
70.0
61.5
Simple yprog
57.5
72.5
55.0
46.3
56.7
42.5
48.8
53.1
54.1
Gram-Schmidt ytar and yprog
33.8
50.0
38.1
16.9
42.5
18.8
30.6
31.2
42.7
R2F (SD3)
89.4
79.4
81.9
80.0
72.5
70.0
58.8
73.8
75.7
ADAPT (Ours)
96.3
88.8
83.8
81.9
79.4
76.9
75.0
82.5
83.1
Table 3: Ablation study on text embedding strategies for the SD3’s sequence representation branch. The top section shows single text embedding methods: using only ytar , using only yprog , and orthogonal decomposition of ytar with yprog . The bottom section shows iterative switching text embedding methods.
Single
Multi
Method
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
Noun-Only
95.6
93.8
79.4
78.1
73.8
76.9
71.3
78.8
81.0
Rare Phrases
96.3
93.1
77.5
78.8
76.9
78.8
75.0
78.8
81.9
All tokens (w ⟨SOS⟩ )
92.5
81.9
76.3
73.8
76.9
72.5
75.6
78.1
78.5
All tokens (w/o ⟨SOS⟩ )
96.3
88.8
83.8
81.9
79.4
76.9
75.0
82.5
83.1
Table 4: Ablation studies examine different attention score extraction strategies.
Single
Multi
Method
Property
Shape
Texture
Action
Complex
Concat
Relation
Complex
Avg
Mean
71.9
87.5
63.7
76.3
75.0
67.5
65.1
79.4
73.3
Cumulative
94.4
80.0
80.6
78.1
81.3
67.5
68.1
80.6
78.8
Individual
96.3
88.8
83.8
81.9
79.4
76.9
75.0
82.5
83.1
Table 5: Quantitative results for different token aggregation methods in attention scoring.
Figure 4: Qualitative comparison on the hyperparameters introduced within the ADAPT framework.
Scores
LAION-aesthetic
PickScore
ImageReward
SD3
6.362±0.128
21.812±0.487
0.959±0.204
R2F
6.395±0.166
22.157±0.301
0.997±0.297
Ours
6.332±0.168
22.188±0.268
1.039±0.260
Table 6: Aesthetic scores of generated images evaluated on RareBench. Our method shows superior PickScore and ImageReward, both capturing human-preference metrics, while maintaining comparable LAION-aesthetic scores.
Figure 6: Attention map visualization for each step. The images are ordered sequentially from left to right across each row, starting from the top.
Figure 7: Further visualization on ADAPT hyperparameters for rare image generation. We vary the pooled embedding scale λpool , the latent manipulation scale λattr , and the attention threshold τs to analyze their effects on generation quality.
Figure 8: Ablation study with and without Attention-based Prompt Scheduling.
< Instructions for Image Preference Evaluation >
In this study, you will be shown a series of image pairs, each accompanied by a text prompt describing a rare or unusual concept. For each pair, please select the image that you believe best matches the given text prompt.
What to Consider
(1) Semantic accuracy: Does the image contain all objects and attributes mentioned in the prompt?
(2) Visual coherence: Is the rare concept visually well-integrated and believable?
Table 7: User Study Instructions.
Figure 9: User study results for each category in the RareBench benchmark. Participants consistently preferred images generated by ADAPT over R2F, indicating stronger text-to-image alignment and rare concept fidelity.
Figure 10: Qualitative analysis of adaptive weighting in PEM with varying parameters (s,p,ϵ) for scaling range, sharpness, and similarity threshold, respectively.
Figure 11: 4-step inference qualitative results of ADAPT combined with FLUX-schnell.
RareBench
Property
Shape
Texture
Action
Complex
FLUX
72.5
68.1
49.3
61.2
73.7
R2F (FLUX)
78.7
75.0
56.8
67.5
68.7
ADAPT (FLUX)
81.9
80.6
61.3
70.0
77.5
Table 8: 4-step inference quantitative results of ADAPT combined with FLUX-schnell.
Figure 12: Illustration of failure cases. For the fair comparison, we use the same reconstructed prompts as R2F.
Models
Property
Shape
Texture
Action
Complex
Avg
SD3.0
49.4
76.3
53.1
71.9
65.0
63.1
Human Generated (R2F)
79.8
68.8
76.3
78.5
-
75.9
Human Generated (Ours)
92.5
83.8
81.9
79.9
-
84.5
LLaMA3 (R2F)
81.9
77.1
76.3
78.8
67.7
76.4
LLaMA3 (Ours)
78.8
80.6
76.3
78.1
70.0
76.8
GPT-4o (R2F)
89.4
79.4
81.9
80.0
72.5
80.6
Table 9: Experiments with different frequent prompt strategies.
Figure 13: Visualization of SAttn on each tokens and images.
Figure 14: Random seeds visualization results of ADAPT on RareBench. Images are generated from eight random prompts with eight random seeds.
< System Prompt >
You are a helper language model for a text-to-image generation program that aims to create images based on input text. The program often struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts.
< User Prompt >
Extract rare concepts from the input text and replace them with relevant yet more frequent ones. Perform the following process step by step:
a. Identify and extract any rare concepts from the provided input text. If the text contains one or more rare concepts, extract them all. If there are no rare concepts present, do not extract any concepts. The extracted rare concepts should not overlap.
b. Given the rare concepts, replace each extracted rare concept with a more frequent concept. Specifically, split each rare concept into the main noun subject and the context, and replace the main noun subject with a more frequent noun subject that is likely to appear in the context of the original rare concept.
Table 10: Full LLM instruction for ADAPT to generate rare-to-frequent concept mappings.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
Rare concept generation focuses on synthesizing customized images conditioned on text prompts that describe objects with unusual attributes. Previous works failed to align the generated images with rare concepts, resulting in incorrect attribute rendering or inconsistent composition of concepts. Such failures, as we observed, stem from the inherent common knowledge bias in the training stage of diffusion models, where objects are strongly associated with their common attributes, making it difficult to break these associations when generating rare concepts. To address such challenges, in this paper, we propose a novel Counterfactual Inference-based Diffusion approach, dubbed CI-Diff. CI-Diff blocks the interference of the model's inherent common knowledge bias and utilizes the Natural Direct Effect to capture the independent influence of the text prompt of rare concepts on image generation so that decoupling the unusual attributes from the rare concepts. To this end, we reformulate the classifier-free guidance mechanism to highlight the atypical attributes. To the best of our knowledge, we are the first to introduce causal inference into the rare concept generation task. Extensive experiments on the RareBench benchmark validate the superiority of CI-Diff over state-of-the-art diffusion models. Our code can be accessed from https://github.com/200204jzy/CI-Diff.
Zhengyuan Jiang, Haipeng Liu, Meng Wang +1
School of Computer Science and Information Engineering, Hefei University of Technology Hefei, China
Text-to-image (T2I) diffusion models have achieved striking progress but still struggle to synthesize rare concepts involving unusual attribute-object pairings, often resulting in concept omission or semantic drift where a dominant entity overwhelms the generation. Tracing these failures to a lack of compositional balance during the denoising trajectory, we propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process. RADIANCE augments pretrained backbones with three modular components: (1) a Compositional Similarity Monitor (CSM) that tracks the emergence of objects and attributes in intermediate latents via CLIP-based feedback; (2) a Bidirectional Scale Controller (BSC) that applies a reactive "restoring force" using positive and negative IP-Adapter scales to rebalance biased trajectories; and (3) a Feedback Guidance Scheduler (FGS) that coordinates these updates across timesteps without additional training. We further extend the framework to multi-object prompts via Delayed Adapter Activation (DAA) and Layer-wise Alternating Guidance (LAG) to prevent premature concept fusion. By overlapping monitoring and denoising through pipelined execution, RADIANCE maintains competitive latency while significantly enhancing the per-sample success rate and effective throughput. Experiments on RareBench and T2I-CompBench demonstrate that RADIANCE consistently enhances compositional alignment and perceptual quality over state-of-the-art baselines.
Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.