Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruning

Yang Deng; David Mould; Paul L. Rosin; Yu-Kun Lai

cs.CVApr 10, 2026

Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruning

Authors: Yang Deng, David Mould, Paul L. Rosin, Yu-Kun Lai

Organizations: Cardiff University, Cardiff, United Kingdom · Carleton University, Ottawa, Canada

Abstract

Existing text-to-image diffusion models, while excelling at subject synthesis, exhibit a persistent foreground bias that treats the background as a passive and under-optimized byproduct. This imbalance compromises global scene coherence and constrains compositional control. To address the limitation, we propose a training-free framework that restructures diffusion sampling to explicitly account for foreground-background interactions. Our approach consists of two key components. First, Dynamic Spatial Guidance introduces a soft, time step dependent gating mechanism that modulates foreground and background attention during the diffusion process, enabling spatially balanced generation. Second, Multi-Path Pruning performs multi-path latent exploration and dynamically filters candidate trajectories using both internal attention statistics and external semantic alignment signals, retaining trajectories that better satisfy object-background constraints. We further develop a benchmark specifically designed to evaluate object-background compositionality. Extensive evaluations across multiple diffusion backbones demonstrate consistent improvements in background coherence and object-background compositional alignment.

Figures & tables

Figure 1: Seed sensitivity of Stable Diffusion XL [ 38 ] and FLUX [ 25 ] . With the same prompt, different random seeds produce distinct layouts and semantics. Each row shows four results from different seeds for one model.

Figure 2: Overview of the proposed framework. We expect a long text prompt as input. The Decomposer Agent parses it into structured components including background and object sentences (

\mathcal{P}_{\text{bg}},\mathcal{P}_{\text{obj}}

) and their corresponding entity sets (

\mathcal{N}_{\text{bg}},\mathcal{N}_{\text{obj}}

). These sets are processed by the Tokenizer and embedded for diffusion conditioning

\mathcal{D}_{\theta}

. During denoising, the Dynamic Spatial Guidance (DSG) Gating module derives a time-varying background soft mask

\widehat{M}_{\text{bg}}

from cross-attention statistics over background tokens, as illustrated in the lower panel. This mask guides the Background Projection (BG) step [Eq. ( 5 )], which softly anchors early latent states

z_{t}

toward a reference background latent

z_{t}^{\text{bg}}

, ensuring global consistency without retraining. Multiple stochastic trajectories are explored in parallel, and the Dynamic Pruning module periodically evaluates each path using attention-tail entropy and CLIP alignment scores to discard incoherent candidates. The bottom subfigure details the computation flow of DSG Gating, including attention aggregation, token grouping, and spatial upsampling for mask fusion.

Figure 3: Dynamic Evolution of the Background Soft Mask ( $\widehat{M}_{\mathrm{bg}}$ ) during Latent Denoising. The mask is computed dynamically at each step by aggregating cross-attention over background tokens ( $\mathcal{T}_{\mathrm{bg}}$ ).

Figure 4: Qualitative comparison across diffusion backbones. Methods are grouped by backbone type, highlighted with colored blocks: green for SD2.1-based models, blue for SDXL/DiT-based models, and gray for FLUX-based models. Each row depicts images generated from the same prompt and random seed.

Figure 5: Ablation study on key components. We validate the necessity of the DSG framework on both SDXL [ 38 ] and FLUX [ 25 ] architectures. (A) w/o Projection: Removing BG Projection causes the model to ignore background keywords ( $\mathcal{P}_{\mathrm{bg}}$ ). (B) w/o DSG Gating: Omitting the DSG soft mask (i.e., using $\widehat{M}_{\mathrm{bg}}\equiv 1$ ) forces a global correction, leading to catastrophic blurring and structural ruin, especially in FLUX [ 25 ] . Spatial gating is mandatory for safe latent modification. (C) w/o Dynamic Pruning: Disabling Online Pruning allows paths with high Attention Entropy to proceed, frequently resulting in Concept Collapse and poor semantic focus.

SD2.1-based
Method	Venue	CLIP_I [ 41 ]	BLIP [ 27 ]	NounRecall	HPSv3 [ 31 ]	ID-Sim [ 13 ]
AttendAndExcite [ 9 ]	SIGGRAPH’23	0.3295	0.4535	0.5567	8.1081	-
TokenCompose [ 54 ]	CVPR’24	0.3258	0.4676	0.5509	6.4276	-
STORM [ 19 ]	CVPR’25	0.3278	0.4586	0.5527	8.2355	-
SD2.1 [ 46 ]	CVPR’22	0.3278	0.4405	0.5326	5.6710	-
OursSD2.1	Ours	0.3364	0.4716	0.5586	6.9645	-

Table 1: Global mean metrics comparison across methods. Bold values indicate the best performance within each group, and underlined values denote the second best.

SD2.1-based models
Method	Full	Background	Object
AttendAndExcite [ 9 ]	0.3472	0.3782	0.4456
TokenCompose [ 54 ]	0.2412	0.2647	0.3706
STORM [ 19 ]	0.3091	0.3697	0.3212
Stable Diffusion 2.1 [ 46 ]	0.2623	0.3934	0.3770
OursSD2.1	0.3121	0.4162	0.3815

Table 2: User study results across all methods. Each participant compared pairs of generated images under three criteria: full description, background description, and main object description. Reported values denote win rates per method. Note that PixArt - alpha and SCG adopt a DiT architecture; and playgroundv2.5 follows the SDXL architecture but is trained from scratch . Bold values indicate the best performance within each group, and underlined values denote the second best.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

SDXL backbone
Method	CLIP_I	BLIP	NounRecall	HPSv3
w/o BGproj	0.3475	0.4827	0.5687	8.3981
w/o Dsg	0.2780	0.3456	0.4903	1.0348
w/o Prune	0.3487	0.4858	0.5718	8.4099
w/o clipScore	0.3456	0.4806	0.5713	8.3277
w/o Entropy	0.3521	0.5066	0.5751	7.9871

Appendix

Table 3: Quantitative ablation of core components on SDXL and FLUX backbones.

Figure 6: Qualitative ablation of the Dynamic Pruning signals on SDXL (left block) and FLUX (right block). For each prompt, we show the full model ( Ours ), the variant without CLIP-based scoring ( w/o clipScore , i.e., $w_{\mathrm{clip}}{=}0$ ), and the variant without Tail Entropy ( w/o Entropy , i.e., $w_{\mathrm{tail}}{=}0$ ). Removing the CLIP term often preserves coarse layout but fails to satisfy fine-grained textual details (e.g., missing secondary objects or incorrect background attributes), while removing the entropy term leads to visually noisy or structurally inconsistent regions due to unstable attention patterns. When both signals are used jointly, the selected trajectories exhibit sharper foregrounds and more faithful, text-aligned backgrounds.

Compositional T2I Generation

Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.

See all 98 papers

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal
Dec 25, 2025Sanghyun Jo, Donghwan Lee, Eunji Jung +2Object Removal Diffusion-Based Image Editing
OGQ, Seoul, Korea · Seoul National University, Seoul, Korea
RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement
Jul 6, 2026Zi-Xiang Ni, Bo-Lun Huang, Teng-Fang Hsiao +2Diffusion Model Guidance T2I Generation
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds
Aug 4, 2026Ning Zhu, An Chen, Mengfei Zhao +4T2I Generation Attention Control in Diffusion Models
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China

Dec 25, 2025J/K move · Enter open · S save

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

Sanghyun Jo, Donghwan Lee, Eunji Jung, Seong Je Oh+1

OGQ, Seoul, Korea · Seoul National University, Seoul, Korea

Object Removal Diffusion-Based Image Editing

Open paper AskPDF arXiv

Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches manipulate the diffusion model's internal self-attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a multimodal large-language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and alignment objectives without explicit attention intervention. As a model-agnostic plug-in applicable to diverse diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.

Abstract

Compositional T2I Generation

Explore similar work

EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds