PASTA: Prompt-Aware Annealed Diffusion Sampling

Ren Ajiki; Ryugo Morita; Stanislav Frolov; Brian Bernhard Moser; Ko Watanabe; Riku Takahashi; Andreas Dengel; Jinjia Zhou

cs.CVOct 23, 2026

PASTA: Prompt-Aware Annealed Diffusion Sampling

Authors: Ren Ajiki, Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Andreas Dengel, Jinjia Zhou

Organizations: Faculty of Science and Engineering, Hosei University, Tokyo, Japan · RPTU Kaiserslautern-Landau & DFKI GmbH, Kaiserslautern, Germany

Abstract

While Diffusion models produce high-quality images, real-world visual design requires intentionally preserving clean regions for headlines, logos, or UI elements. Existing layout-aware text-to-image methods struggle to strictly preserve reserved regions, often introducing unwanted content that complicates downstream editing. Existing approaches either manipulate attention to displace content or suppress conditioning via noise-level interventions. However, these methods sacrifice visual coherence, producing flat regions or noticeable style inconsistencies between reserved and non-reserved areas. To address this limitation, we propose a Prompt-Aware Shared-Latent Training-Free Annealed Diffusion Sampling (PASTA), a training-free framework that enforces reserved-region constraints by selectively applying semantic guidance to object-allowed regions during sampling. PASTA operates on a shared latent trajectory, where full-prompt and context-prompt guidance are fused through mask-guided sampling. An annealed schedule suppresses semantic leakage during early denoising stages while gradually restoring visual consistency in later stages. Extensive experiments demonstrate that PASTA improves reserved-region fidelity while preserving overall image realism and perceptual consistency.

Figures & tables

Figure 1: Red words in the prompt column indicate the context prompt

P_{\mathrm{ctx}}

, and yellow dashed boxes denote the reserved regions. TextCenGen [ 14 ] causes foreground content to overlap with the placed logo-style text element, while SAWNA [ 20 ] produces a flat and visually inconsistent reserved region. PASTA keeps the reserved region clean while maintaining a coherent background.

Figure 2: Overview of PASTA. Given a full prompt $P_{\mathrm{full}}$ and a reserved-region mask $M$ , PASTA computes guidance residuals from shared-latent noise predictions under full-prompt, context-prompt, and negative-prompt conditions. The residuals are spatially fused to suppress foreground-object guidance inside the reserved region, while an annealed schedule controls the context-prompt contribution in the object-allowed region.

Figure 3: Qualitative comparison under small- and large-mask settings. Rows 1–4 show small-mask examples, while Rows 5–8 show large-mask examples. Within each mask setting, the first two rows show SynPrompt examples and the last two rows show P2P examples. Red words in the prompt column indicate the context prompt $P_{\mathrm{ctx}}$ , and yellow dashed boxes denote reserved regions. TextCenGen can slightly adjust object placement under small-mask settings but often fails to satisfy large-mask constraints. SAWNA suppresses object generation inside the reserved regions, but frequently produces flat or near-monochrome backgrounds with visible discontinuities. PASTA preserves clean reserved regions while maintaining natural and coherent backgrounds across both mask sizes.

P2P Prompts
Method	Setting	CLIP $\uparrow$	HPS $\uparrow$	Intr. $\downarrow$
SDXL	Small	33.28	0.291	0.278
TextCenGen [ 14 ]	Small	33.06	0.291	0.287
SAWNA [ 20 ]	Small	33.25	0.284	0.148
Ours	Small	33.08	0.284	0.098
SDXL	Large	33.28	0.291	0.223

Table 1: Quantitative comparison under small and large reserved-region masks. Higher is better for CLIP and HPS, while lower is better for Intrusion Rate.

Setting	CLIP $\uparrow$	HPS $\uparrow$	Intrusion $\downarrow$
Ours (w/o annealed weight $\alpha$ )	33.035	0.2899	0.130
Ours (w/ annealed weight $\alpha$ )	32.263	0.2770	0.038

Table 2: Ablation study on the effect of the annealed weight

\alpha

on reserved-region preservation. Experiments are conducted using P2P prompts and the large-mask setting. Higher is better for CLIP and HPS, while lower is better for Intrusion Rate.

Figure 4: Qualitative ablation of the annealed blending schedule. All results use the same prompt, mask, and random seed. Red words in the prompt column indicate the context prompt $P_{\mathrm{ctx}}$ , and yellow dashed boxes denote reserved regions. Without $\alpha$ or with a weak constant weight $\alpha=1$ , distracting foreground or vegetation-like structures remain inside the reserved region. A strong constant weight $\alpha=5$ produces a cleaner region but over-suppresses the foreground object. The proposed annealed schedule better balances reserved-region cleanliness and foreground fidelity.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
Mar 19, 2026Kwanyoung Lee, SeungJu Cha, Yebin Ahn +3T2I Generation T2I Diffusion Models
Hanyang University, South Korea
Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformers
Mar 6, 2026Ruidong Chen, Yancheng Bai, Xuanpu Zhang +6Controllable Image Generation Compositional T2I Generation
Tianjin University · Independent Researcher
Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection
Mar 13, 2026Yunzhuo Chen, Jordan Vice, Naveed Akhtar +2T2I Diffusion Models T2I Generation
The University of Western Australia, Perth, Australia · The University of Melbourne, Melbourne, Australia · Curtin University, Perth, Australia

Mar 19, 2026J/K move · Enter open · S save

Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation

Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh+2

Hanyang University, South Korea

T2I Generation T2I Diffusion Models Diffusion Model Guidance Diffusion-Based Image Editing

Open paper AskPDF arXiv

Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.

Abstract

T2I Generation

Explore similar work

Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation

Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformers

Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection