Existing text-to-image diffusion models, while excelling at subject synthesis, exhibit a persistent foreground bias that treats the background as a passive and under-optimized byproduct. This imbalance compromises global scene coherence and constrains compositional control. To address the limitation, we propose a training-free framework that restructures diffusion sampling to explicitly account for foreground-background interactions. Our approach consists of two key components. First, Dynamic Spatial Guidance introduces a soft, time step dependent gating mechanism that modulates foreground and background attention during the diffusion process, enabling spatially balanced generation. Second, Multi-Path Pruning performs multi-path latent exploration and dynamically filters candidate trajectories using both internal attention statistics and external semantic alignment signals, retaining trajectories that better satisfy object-background constraints. We further develop a benchmark specifically designed to evaluate object-background compositionality. Extensive evaluations across multiple diffusion backbones demonstrate consistent improvements in background coherence and object-background compositional alignment.
Figures & tables
Figure 1: Seed sensitivity of Stable Diffusion XL [ 38 ] and FLUX [ 25 ] . With the same prompt, different random seeds produce distinct layouts and semantics. Each row shows four results from different seeds for one model.
Figure 2: Overview of the proposed framework. We expect a long text prompt as input. The Decomposer Agent parses it into structured components including background and object sentences ( Pbg,Pobj ) and their corresponding entity sets ( Nbg,Nobj ). These sets are processed by the Tokenizer and embedded for diffusion conditioning Dθ . During denoising, the Dynamic Spatial Guidance (DSG) Gating module derives a time-varying background soft mask Mbg from cross-attention statistics over background tokens, as illustrated in the lower panel. This mask guides the Background Projection (BG) step [Eq. ( 5 )], which softly anchors early latent states zt toward a reference background latent ztbg , ensuring global consistency without retraining. Multiple stochastic trajectories are explored in parallel, and the Dynamic Pruning module periodically evaluates each path using attention-tail entropy and CLIP alignment scores to discard incoherent candidates. The bottom subfigure details the computation flow of DSG Gating, including attention aggregation, token grouping, and spatial upsampling for mask fusion.
Figure 3: Dynamic Evolution of the Background Soft Mask ( Mbg ) during Latent Denoising. The mask is computed dynamically at each step by aggregating cross-attention over background tokens ( Tbg ).
Figure 4: Qualitative comparison across diffusion backbones. Methods are grouped by backbone type, highlighted with colored blocks: green for SD2.1-based models, blue for SDXL/DiT-based models, and gray for FLUX-based models. Each row depicts images generated from the same prompt and random seed.
Figure 5: Ablation study on key components. We validate the necessity of the DSG framework on both SDXL [ 38 ] and FLUX [ 25 ] architectures. (A) w/o Projection: Removing BG Projection causes the model to ignore background keywords ( Pbg ). (B) w/o DSG Gating: Omitting the DSG soft mask (i.e., using Mbg≡1 ) forces a global correction, leading to catastrophic blurring and structural ruin, especially in FLUX [ 25 ] . Spatial gating is mandatory for safe latent modification. (C) w/o Dynamic Pruning: Disabling Online Pruning allows paths with high Attention Entropy to proceed, frequently resulting in Concept Collapse and poor semantic focus.
Method
Venue
CLIP_I [ 41 ]
BLIP [ 27 ]
NounRecall
HPSv3 [ 31 ]
ID-Sim [ 13 ]
SD2.1-based
AttendAndExcite [ 9 ]
SIGGRAPH’23
0.3295
0.4535
0.5567
8.1081
-
TokenCompose [ 54 ]
CVPR’24
0.3258
0.4676
0.5509
6.4276
-
STORM [ 19 ]
CVPR’25
0.3278
0.4586
0.5527
8.2355
-
SD2.1 [ 46 ]
CVPR’22
0.3278
0.4405
0.5326
5.6710
-
OursSD2.1
Ours
0.3364
0.4716
0.5586
6.9645
-
Table 1: Global mean metrics comparison across methods. Bold values indicate the best performance within each group, and underlined values denote the second best.
Method
Full
Background
Object
SD2.1-based models
AttendAndExcite [ 9 ]
0.3472
0.3782
0.4456
TokenCompose [ 54 ]
0.2412
0.2647
0.3706
STORM [ 19 ]
0.3091
0.3697
0.3212
Stable Diffusion 2.1 [ 46 ]
0.2623
0.3934
0.3770
OursSD2.1
0.3121
0.4162
0.3815
Table 2: User study results across all methods. Each participant compared pairs of generated images under three criteria: full description, background description, and main object description. Reported values denote win rates per method. Note that PixArt - alpha and SCG adopt a DiT architecture; and playgroundv2.5 follows the SDXL architecture but is trained from scratch . Bold values indicate the best performance within each group, and underlined values denote the second best.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
CLIP_I
BLIP
NounRecall
HPSv3
SDXL backbone
w/o BGproj
0.3475
0.4827
0.5687
8.3981
w/o Dsg
0.2780
0.3456
0.4903
1.0348
w/o Prune
0.3487
0.4858
0.5718
8.4099
w/o clipScore
0.3456
0.4806
0.5713
8.3277
w/o Entropy
0.3521
0.5066
0.5751
7.9871
Appendix
Table 3: Quantitative ablation of core components on SDXL and FLUX backbones.
Figure 6: Qualitative ablation of the Dynamic Pruning signals on SDXL (left block) and FLUX (right block). For each prompt, we show the full model ( Ours ), the variant without CLIP-based scoring ( w/o clipScore , i.e., wclip=0 ), and the variant without Tail Entropy ( w/o Entropy , i.e., wtail=0 ). Removing the CLIP term often preserves coarse layout but fails to satisfy fine-grained textual details (e.g., missing secondary objects or incorrect background attributes), while removing the entropy term leads to visually noisy or structurally inconsistent regions due to unstable attention patterns. When both signals are used jointly, the selected trajectories exhibit sharper foregrounds and more faithful, text-aligned backgrounds.
Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.
Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches manipulate the diffusion model's internal self-attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a multimodal large-language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives without explicit attention intervention. As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.
Sanghyun Jo, Donghwan Lee, Eunji Jung +2
OGQ, Seoul, Korea · Seoul National University, Seoul, Korea
Text-to-image (T2I) diffusion models have achieved striking progress but still struggle to synthesize rare concepts involving unusual attribute-object pairings, often resulting in concept omission or semantic drift where a dominant entity overwhelms the generation. Tracing these failures to a lack of compositional balance during the denoising trajectory, we propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process. RADIANCE augments pretrained backbones with three modular components: (1) a Compositional Similarity Monitor (CSM) that tracks the emergence of objects and attributes in intermediate latents via CLIP-based feedback; (2) a Bidirectional Scale Controller (BSC) that applies a reactive "restoring force" using positive and negative IP-Adapter scales to rebalance biased trajectories; and (3) a Feedback Guidance Scheduler (FGS) that coordinates these updates across timesteps without additional training. We further extend the framework to multi-object prompts via Delayed Adapter Activation (DAA) and Layer-wise Alternating Guidance (LAG) to prevent premature concept fusion. By overlapping monitoring and denoising through pipelined execution, RADIANCE maintains competitive latency while significantly enhancing the per-sample success rate and effective throughput. Experiments on RareBench and T2I-CompBench demonstrate that RADIANCE consistently enhances compositional alignment and perceptual quality over state-of-the-art baselines.
Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Text-to-image diffusion models can generate individual concepts well, but they often omit or merge concepts incorrectly with multiple concepts. We trace these failures to an early coordination bottleneck: before denoising begins, prompt-conditioned attention may allocate different concepts to strongly overlapping spatial support, which can keep their attention coupled as denoising proceeds. This observation motivates treating compositional generation as a boundary-condition problem rather than repeatedly controlling the evolving trajectory. To this end, we propose Rectify-then-Diffuse (RTD), a training-free framework that rectifies the initial allocation once before standard denoising. Firstly, we propose Soft-Overlap Disentanglement (SOD), which converts normalized overlap between pilot concept maps into a differentiable and layout-agnostic separation objective. Secondly, we introduce Isotropic Gradient Rectification (IGR), which normalizes the SOD gradient and applies a bounded latent displacement with a consistent scale across prompts and initializations. Extensive experiments show that RTD achieves state-of-the-art compositional fidelity and robust gains. On the AE-Bench object pair subset, RTD improves BLIP-VQA by 45.8% and ImageReward by 19.6% over CO3 while running 2.3× faster. Code will be released at https://github.com/Z-yiwei/rectify-then-diffuse
Ning Zhu, An Chen, Mengfei Zhao +4
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China