ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation

Kwanyoung Lee; Hyunwoo Oh; SeungJu Cha; Sungho Koh; Dong-Jin Kim

cs.CVMar 19, 2026

ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation

Authors: Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim

Organizations: Hanyang University, South Korea

Abstract

Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.

Figures & tables

Figure 2: Overview of our ADAPT framework. (a) Our framework introduces three complementary zero-shot control modules in the Stable Diffusion 3 (SD3) architecture. Adaptive Prompt Scheduling (APS) determines optimal stop points $S^{i}$ based on each token’s spatial attention map score $\mathcal{S}_{\text{Attn}}$ at each step; Pooled Embedding Manipulation (PEM) operates on the CLIP’s pooled text embedding prior to modulation in the SD3 pipeline; Latent Space Manipulation (LSM) is applied at the feature level directly within the transformer block after attention computation. (b) The overview of attention scoring. We compute Top-k attention response scores $s^{(k)}$ to determine when each token’s semantic content has been sufficiently established. (c) Visualization of adaptive prompt scheduling between $y_{\text{prog}}$ and target prompt $y_{\text{tar}}$ based on stop points $S^{i}$ .

	Single Object					Multi Objects
Models	Property	Shape	Texture	Action	Complex	Concat	Relation	Complex	Avg
SD1.5 [ 20 ]	55.0	38.8	33.8	23.1	36.9	23.1	24.4	36.3	33.9
SDXL [ 16 ]	60.0	56.9	71.3	47.5	58.1	39.4	35.0	47.5	52.0
PixArt [ 4 ]	49.4	58.8	76.9	56.3	63.1	35.6	30.0	48.1	52.3
SD3.0 [ 6 ]	49.4	76.3	53.1	71.9	65.0	55.0	51.2	70.0	61.5
FLUX [ 1 ]	58.1	71.9	47.5	52.5	60.0	55.0	48.1	70.3	57.9

Table 1: Text-to-image alignment performances in the RareBench with other baselines with GPT-4o based evaluation. Best values are notated with bold , second-best with underlined .

	Single					Multi
Method	Property	Shape	Texture	Action	Complex	Concat	Relation	Complex	Avg
R2F (SD3)	89.4	79.4	81.9	80.0	72.5	70.0	58.8	73.8	75.7
PEM (w/o Adaptive)	90.0	84.4	78.8	71.9	76.9	71.9	73.8	79.4	78.4
PEM	92.5	91.3	78.8	69.4	78.1	70.6	77.5	80.0	79.8
PEM + LSM	92.5	91.3	80.6	71.9	77.5	73.1	76.9	79.4	80.4
PEM + APS	96.3	88.8	80.0	77.5	78.1	74.4	71.3	79.4	80.7

Table 2: Ablation studies demonstrating the incremental contribution of each proposed component (PEM, LSM, and APS) to text-to-image alignment performance across different evaluation categories on the RareBench dataset.

Figure 3: Qualitative comparison between previous methods and our method across rare concept prompts from RareBench. Our approach significantly improves upon R2F in a zero-shot setting, demonstrating superior alignment between image and text for a wide range of challenging attributes and compositions. All samples are generated with the same random seed (42) for fair comparison.

	Single					Multi
Method	Property	Shape	Texture	Action	Complex	Concat	Relation	Complex	Avg
Simple $y_{\text{tar}}$	49.4	76.3	53.1	71.9	65.0	55.0	51.2	70.0	61.5
Simple $y_{\text{prog}}$	57.5	72.5	55.0	46.3	56.7	42.5	48.8	53.1	54.1
Gram-Schmidt $y_{\text{tar}}$ and $y_{\text{prog}}$	33.8	50.0	38.1	16.9	42.5	18.8	30.6	31.2	42.7
R2F (SD3)	89.4	79.4	81.9	80.0	72.5	70.0	58.8	73.8	75.7
ADAPT (Ours)	96.3	88.8	83.8	81.9	79.4	76.9	75.0	82.5	83.1

Table 3: Ablation study on text embedding strategies for the SD3’s sequence representation branch. The top section shows single text embedding methods: using only

y_{\text{tar}}

, using only

y_{\text{prog}}

, and orthogonal decomposition of

y_{\text{tar}}

with

y_{\text{prog}}

. The bottom section shows iterative switching text embedding methods.

	Single					Multi
Method	Property	Shape	Texture	Action	Complex	Concat	Relation	Complex	Avg
Noun-Only	95.6	93.8	79.4	78.1	73.8	76.9	71.3	78.8	81.0
Rare Phrases	96.3	93.1	77.5	78.8	76.9	78.8	75.0	78.8	81.9
All tokens (w $\langle SOS\rangle$ )	92.5	81.9	76.3	73.8	76.9	72.5	75.6	78.1	78.5
All tokens (w/o $\langle SOS\rangle$ )	96.3	88.8	83.8	81.9	79.4	76.9	75.0	82.5	83.1

Table 4: Ablation studies examine different attention score extraction strategies.

	Single					Multi
Method	Property	Shape	Texture	Action	Complex	Concat	Relation	Complex	Avg
Mean	71.9	87.5	63.7	76.3	75.0	67.5	65.1	79.4	73.3
Cumulative	94.4	80.0	80.6	78.1	81.3	67.5	68.1	80.6	78.8
Individual	96.3	88.8	83.8	81.9	79.4	76.9	75.0	82.5	83.1

Table 5: Quantitative results for different token aggregation methods in attention scoring.

Figure 4: Qualitative comparison on the hyperparameters introduced within the ADAPT framework.

Scores	LAION-aesthetic	PickScore	ImageReward
SD3	6.362±0.128	21.812±0.487	0.959±0.204
R2F	6.395±0.166	22.157±0.301	0.997±0.297
Ours	6.332±0.168	22.188±0.268	1.039±0.260

Table 6: Aesthetic scores of generated images evaluated on RareBench. Our method shows superior PickScore and ImageReward, both capturing human-preference metrics, while maintaining comparable LAION-aesthetic scores.

Figure 5: Attention response score $\mathcal{S}_{\text{Attn}}$ of target prompt “A horned pelican”.

Figure 6: Attention map visualization for each step. The images are ordered sequentially from left to right across each row, starting from the top.

Figure 7: Further visualization on ADAPT hyperparameters for rare image generation. We vary the pooled embedding scale $\lambda_{\text{pool}}$ , the latent manipulation scale $\lambda_{\text{attr}}$ , and the attention threshold $\tau_{s}$ to analyze their effects on generation quality.

Figure 8: Ablation study with and without Attention-based Prompt Scheduling.

<

Instructions for Image Preference Evaluation

>

In this study, you will be shown a series of image pairs, each accompanied by a text prompt describing a rare or unusual concept. For each pair, please select the image that you believe best matches the given text prompt.

What to Consider

(1) Semantic accuracy: Does the image contain all objects and attributes mentioned in the prompt?

(2) Visual coherence: Is the rare concept visually well-integrated and believable?

Table 7: User Study Instructions.

Figure 9: User study results for each category in the RareBench benchmark. Participants consistently preferred images generated by ADAPT over R2F, indicating stronger text-to-image alignment and rare concept fidelity.

Figure 10: Qualitative analysis of adaptive weighting in PEM with varying parameters $(s,p,\epsilon)$ for scaling range, sharpness, and similarity threshold, respectively.

Figure 11: 4-step inference qualitative results of ADAPT combined with FLUX-schnell.

RareBench	Property	Shape	Texture	Action	Complex
FLUX	72.5	68.1	49.3	61.2	73.7
R2F (FLUX)	78.7	75.0	56.8	67.5	68.7
ADAPT (FLUX)	81.9	80.6	61.3	70.0	77.5

Table 8: 4-step inference quantitative results of ADAPT combined with FLUX-schnell.

Figure 12: Illustration of failure cases. For the fair comparison, we use the same reconstructed prompts as R2F.

Models	Property	Shape	Texture	Action	Complex	Avg
SD3.0	49.4	76.3	53.1	71.9	65.0	63.1
Human Generated (R2F)	79.8	68.8	76.3	78.5	-	75.9
Human Generated (Ours)	92.5	83.8	81.9	79.9	-	84.5
LLaMA3 (R2F)	81.9	77.1	76.3	78.8	67.7	76.4
LLaMA3 (Ours)	78.8	80.6	76.3	78.1	70.0	76.8
GPT-4o (R2F)	89.4	79.4	81.9	80.0	72.5	80.6

Table 9: Experiments with different frequent prompt strategies.

Figure 13: Visualization of $S_{\text{Attn}}$ on each tokens and images.

Figure 14: Random seeds visualization results of ADAPT on RareBench. Images are generated from eight random prompts with eight random seeds.

<

System Prompt

>

You are a helper language model for a text-to-image generation program that aims to create images based on input text. The program often struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts.

<

User Prompt

>

Extract rare concepts from the input text and replace them with relevant yet more frequent ones. Perform the following process step by step:

a. Identify and extract any rare concepts from the provided input text. If the text contains one or more rare concepts, extract them all. If there are no rare concepts present, do not extract any concepts. The extracted rare concepts should not overlap.

b. Given the rare concepts, replace each extracted rare concept with a more frequent concept. Specifically, split each rare concept into the main noun subject and the context, and replace the main noun subject with a more frequent noun subject that is likely to appear in the context of the original rare concept.

Table 10: Full LLM instruction for ADAPT to generate rare-to-frequent concept mappings.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Rare Concept Generation via Counterfactual Inference in Diffusion Models
Jul 16, 2026Zhengyuan Jiang, Haipeng Liu, Meng Wang +1Diffusion Model Guidance T2I Generation
School of Computer Science and Information Engineering, Hefei University of Technology Hefei, China
RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement
Jul 6, 2026Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao +2Diffusion Model Guidance T2I Generation
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
Mar 19, 2026Kwanyoung Lee, SeungJu Cha, Yebin Ahn +3T2I Generation T2I Diffusion Models
Hanyang University, South Korea

Jul 16, 2026J/K move · Enter open · S save

Rare Concept Generation via Counterfactual Inference in Diffusion Models

Zhengyuan Jiang, Haipeng Liu, Meng Wang, Yang Wang

School of Computer Science and Information Engineering, Hefei University of Technology Hefei, China

Diffusion Model Guidance T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Rare concept generation focuses on synthesizing customized images conditioned on text prompts that describe objects with unusual attributes. Previous works failed to align the generated images with rare concepts, resulting in incorrect attribute rendering or inconsistent composition of concepts. Such failures, as we observed, stem from the inherent common knowledge bias in the training stage of diffusion models, where objects are strongly associated with their common attributes, making it difficult to break these associations when generating rare concepts. To address such challenges, in this paper, we propose a novel Counterfactual Inference-based Diffusion approach, dubbed CI-Diff. CI-Diff blocks the interference of the model's inherent common knowledge bias and utilizes the Natural Direct Effect to capture the independent influence of the text prompt of rare concepts on image generation so that decoupling the unusual attributes from the rare concepts. To this end, we reformulate the classifier-free guidance mechanism to highlight the atypical attributes. To the best of our knowledge, we are the first to introduce causal inference into the rare concept generation task. Extensive experiments on the RareBench benchmark validate the superiority of CI-Diff over state-of-the-art diffusion models. Our code can be accessed from https://github.com/200204jzy/CI-Diff.

Abstract

T2I Generation

Explore similar work

Rare Concept Generation via Counterfactual Inference in Diffusion Models

RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement

Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation