Where to Look Is Not How to Fix: Pre-Denoising Diagnostics and Modality-Dependent Control in Diffusion Composition

Fangzheng Wu; Brian Summa

cs.CVOct 2, 2026

Where to Look Is Not How to Fix: Pre-Denoising Diagnostics and Modality-Dependent Control in Diffusion Composition

Authors: Fangzheng Wu, Brian Summa

Organizations: Department of Computer Science, Tulane University, New Orleans, LA 70118

Abstract

Understanding compositional failures in text-to-image diffusion requires identifying both where stress is detectable and how intervention changes the output. We study these questions through a controlled anchor--stress protocol that jointly evaluates text-encoder diagnostics and denoiser interventions. We introduce a text-only Compositional Stress Index (CSI), which separates common from rare compositions across SD1.5, SDXL, and the SD3 text path and provides an upstream diagnostic coordinate. A matched six-prompt localization study links intervention location to distinct outcomes: residual-minimizing embedding adapters improve representation fit, while downstream cross-attention intervention increases color hit rate (CHR) by 0.0272. Across SD1.5 and SDXL denoiser blocks, the largest positive signed diagnostic-accessibility mean occurs at the deep encoder, whereas selective boost has its largest positive mean CHR response at decoder blocks. Selective subtraction and broad ablation reveal further modality- and architecture-dependent responses, including a substantial CHR decrease when SDXL decoder cross-attention is broadly ablated. We find a diagnosis-control dissociation under our controlled attribute-object composition setting: compositional defects are diagnosable before denoising, but the representation coordinate that exposes risk is not necessarily the coordinate or modality that improves generation.

Figures & tables

Pipeline site	Diagnostic readout	Targeted edits	Broad ablation
Text encoder	Residual signal	Residual reduction	—
Deep encoder	Positive DA peak	Small CHR changes	Architecture-dependent
Decoder	Smaller signed DA	Peak mean boost CE	SDXL U0: decrease

Table 1: Diagnostic readouts and observed intervention responses across pipeline sites. Entries summarize measured profiles rather than significance decisions; effect sizes, uncertainty, and statistical tests are reported in Sections 5 – 6 . DA peaks refer to signed means.

Figure 1: Prompt-level CSI distributions for anchor and stress prompts across SD1.5, SDXL, and SD3. Values are displayed on a MAD-normalized symlog scale. Although absolute magnitudes are not directly comparable across models, stress prompts consistently occupy a much higher CSI regime than anchor prompts. This visualization complements ROC-AUC by showing the magnitude of the anchor–stress separation.

Figure 2: Cross-model CSI decile curves. Probability of stress membership increases monotonically with CSI decile; SD1.5, SDXL, and SD3 collapse to the same transition pattern. ROC-AUC

{=}1.0

in all three models.

Figure 3: DA (blue) and CE under boost (green) across cross-attention block groups. The plotted DA and CE profiles are historical; updated phase1 means place the largest positive DA at D2 and boost CE at U0/U1. The largest positive signed DA mean occurs at D2. The retained plot uses historical runs and intervals; phase1 prompt-level CE results are reported in Table 6 .

Figure 4: Same-site (D2, purple) vs. relocated (decoder, green) under boost. Updated prompt-level decoder-minus-D2 tests give

p{=}0.0028

(SDXL) and

p{=}0.1882

(SD1.5). Plotted intervals are historical; updated inference is reported in Appendix F .

Panel of Figure 5 — Figure 5: Same-site (D2) vs. relocated (decoder) boost. D2 boost produces no visible change or degradation; decoder boost increases CHR in these examples. “Gold guitar”: CHR 0.125 $\to$ 0.114 $\to$ 0.244 . “Blue apple”: 0.239 $\to$ 0.224 $\to$ 0.372 . “Red horse”: 0.148 $\to$ 0.160 $\to$ 0.329 .

Figure 6: CE under boost (green) and zeroing (orange) at each block group. The retained plot shows historical estimates; updated prompt-level inference is in Appendix G . Broad zeroing substantially reduces CHR at SDXL U0; absence of a significant D2 effect does not establish tolerance.

Figure 7: Zeroing fragility curve. Colors indicate the sign of the estimated CHR change, not corrected significance or functional necessity. SDXL has a large negative response at U0; SD1.5 at D0/D1. Historical plot annotations require the updated inference in Table 8 .

		Selective additive	Selective subtractive	Broad subtractive
Model	Block	(boost)	( attr_zero )	(zero)
SDXL	D2	$-0.0013$ (ns)	$-0.0011$ (ns)	$+0.0054$ (ns)
SDXL	U0	$+0.0061$ ( $p{=}0.0168$ )	$-0.0002$ (ns)	$\mathbf{-0.1338}$ ( $q{=}0.0010$ )
SD1.5	D2	$+0.0021$ (ns)	$-0.0029$ (ns)	$+0.0577$ ( $q{=}0.0623$ )
SD1.5	U1	$+0.0084$ ( $p{=}0.0384$ )	$-0.0031$ (ns)	$+0.0207$ (ns)

Table 2: CE under three modalities at D2 and decoder (both models). Bold: FDR

q{<}0.05

. Boost

p

-values are uncorrected; FDR across block groups does not reach significance (see text).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Selective (token-level)	Broad (block/pool-wide)
Embedding additive	Token adapter (MSE)	Pooled adapter
Denoiser additive	Boost (attr $\times$ obj mask)	Untested (identity)
Denoiser subtractive	Attr_zero (attr $\times$ obj mask)	Zero (uniform $1/k$ )

Appendix

Table 3: Selectivity

\times

sign intervention matrix. Three denoiser cells and two embedding cells are tested; broad additive is a no-op under post-softmax renormalization.

Model	Condition	DA(D2)	FDR $q$
SDXL	Boost baseline	$+0.0039$	$0.0174$
SDXL	Zero baseline	$+0.0035$	$0.0332$
SDXL	Attr_zero baseline	$+0.0039$	$0.0174$
SD1.5	Boost baseline	$+0.0282$	$0.00028$
SD1.5	Zero baseline	$+0.0296$	$0.00028$
SD1.5	Attr_zero baseline	$+0.0282$	$0.00028$

Appendix

Table 4: DA at D2 across all experimental conditions. D2 is the DA peak in every condition.

Model	Comparison	$\Delta$ CHR	$p$	Sig.?
SDXL	Same-site (D2) vs baseline	$-0.0013$	$0.4093$	No
	Relocated (U0) vs baseline	$+0.0061$	$0.0168$	Yes
	Relocated vs same-site	$\mathbf{+0.0074}$	$\mathbf{0.0028}$	Yes
SD1.5	Same-site (D2) vs baseline	$+0.0021$	$0.4819$	No
	Relocated (U1) vs baseline	$+0.0084$	$0.0384$	Yes
	Relocated vs same-site	$+0.0063$	$0.1882$	No

Appendix

Table 5: Same-site (D2) vs relocated (decoder) under boost. Tests use 52 prompt means after pairing five seeds; displayed significance is nominal, before multiplicity correction.

Block	SDXL CE	SDXL $p$	SD1.5 CE	SD1.5 $p$
D0	–	–	$-0.0031$	0.1854
D1	$-0.0002$	0.8826	$-0.0041$	0.0182
D2	$-0.0013$	0.4093	$+0.0021$	0.4819
M	$+0.0018$	0.1242	$-0.0015$	0.2763
U0	$+0.0061$	0.0168	–	–
U1	$+0.0011$	0.1788	$+0.0084$	0.0384

Appendix

Table 6: CE

{}_{\text{boost}}

per block. Columns show raw

p

-values. No block survives FDR correction; decoder peaks (U0, U1) are nominally significant with consistent cross-model direction.

Block	SDXL CE	SDXL FDR $q$	SD1.5 CE	SD1.5 FDR $q$
D0	–	–	$\mathbf{-0.0050}$	0.0014
D1	$+0.0003$	0.8514	$-0.0010$	0.4669
D2	$-0.0011$	0.8514	$-0.0029$	0.0602
M	$-0.0003$	0.8514	$-0.0022$	0.1225
U0	$-0.0002$	0.8514	–	–
U1	$+0.0007$	0.8514	$-0.0031$	0.0970

Appendix

Table 7: CE

{}_{\text{attr\_zero}}

per block with prompt-level BH

q

. SD1.5 D0 has a small significant negative effect; no block has a significant positive effect.

Block	SDXL CE	SDXL $p$	SD1.5 CE	SD1.5 $p$
D0	–	–	$\mathbf{-0.2199}$	0.0002
D1	$\mathbf{+0.0200}$	0.0132	$\mathbf{-0.1731}$	0.0002
D2	$+0.0054$	0.6619	$+0.0577$	0.0356
M	$-0.0037$	0.4395	$+0.0123$	0.3511
U0	$\mathbf{-0.1338}$	0.0002	–	–
U1	$+0.0008$	0.9126	$+0.0207$	0.4597

Appendix

Table 8: CE

{}_{\text{zero}}

per block. Bold: FDR

q<0.05

. Columns show raw

p

-values; boldface indicates FDR-corrected significance.

Block	SDXL	SD1.5
D0	0	4
D1	4	4
D2	20	4
M	10	2
U0	30	0
U1	6	6

Appendix

Table 9: Module inventories per block group. SDXL counts cross-attention only; SD1.5 retains the legacy all-patched attention inventory. The two columns use different counting conventions and must not be compared as intervention exposure.

Phase	D2 DA	D2 CE (boost)	U0 CE (boost)
Early	$+0.016$	$-0.010$	$+0.008$
Mid	$+0.013$	$+0.001$	$+0.002$
Late	$+0.008$	$-0.000$	$+0.001$

Appendix

Table 10: Per-phase DA and CE at D2 (SDXL, 100 prompts

\times

3 seeds).

Model	Block	$d_{z}$ (per-prompt)	$n_{\text{prompt}}$	Power	$n$ for 80% power
SDXL	U0	$0.345$	$52$	$0.685$	$68$
SD1.5	U1	$0.297$	$52$	$0.557$	$91$

Appendix

Table 11: Effect size and power for decoder boost at the full color-stress subset.

Group	Anchor	Stress
B001	a studio photo of a black cat	a studio photo of a glass cat
B005	a studio photo of a yellow banana	a studio photo of a metallic banana
B028	a studio photo of a green pear	a studio photo of a blue pear
B045	a studio photo of a brown guitar	a studio photo of a gold guitar

Appendix

Table 12: Representative anchor–stress pairs.

$\alpha$	D2 CE	U0 CE
0.00	$-0.0006$	$+0.0002$
0.25	$+0.0003$	$-0.0006$
0.50	$+0.0008$	$+0.0005$
0.75	$+0.0011$	$+0.0002$
Reference: boost U0 $=+0.006$ ( $p{=}0.0168$ )
Reference: zeroing D2 $=+0.058$ (SD1.5, $q{=}0.0623$ )

Appendix

Table 13: Fractional attenuation CE at D2 and U0 (SDXL,

n{=}156

color-prompt pairs per level). No attenuation level produces practically meaningful effect sizes.

Experiment	GPU-hours (approx.)
CSI pipeline (cross-model factorization, SD1.5+SDXL+SD3)	24
Embedding repair (pooled + token adapters, rank sweep)	12
Output-side BLIP scoring	4
Per-block boost sweep (432 $\times$ 5 seeds $\times$ 2 models)	43
Per-block zeroing sweep (432 $\times$ 3 seeds $\times$ 2 models)	20
Attr_zero full-scale (432 $\times$ 5 seeds $\times$ 2 models)	36

Appendix

Table 14: Compute budget by experiment phase.

Compositional T2I Generation

Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.

See all 99 papers

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds
Aug 4, 2026Ning Zhu, An Chen, Mengfei Zhao +4T2I Generation Attention Control in Diffusion Models
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Oct 7, 2026Arshia Hemmat, Amirhossein Vahidi, Amitis Shidani +4Compositional T2I Generation T2I Diffusion Models
Wellcome Sanger Institute · Cambridge Stem Cell Institute, University of Cambridge · Cambridge Centre for AI in Medicine, University of Cambridge +6
Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time
Jul 29, 2026Xinyi Wang, Yuyang Huang, Yalin Su +4Diffusion Model Guidance Text-Image Alignment
Shanghai Jiao Tong University, Shanghai, China

Aug 4, 2026J/K move · Enter open · S save

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

Ning Zhu, An Chen, Mengfei Zhao, Juntao Xu+3

Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China

T2I Generation Attention Control in Diffusion Models Compositional T2I Generation T2I Diffusion Models Diffusion-Based Denoising

Open paper AskPDF arXiv

Text-to-image diffusion models can generate individual concepts well, but they often omit or merge concepts incorrectly with multiple concepts. We trace these failures to an early coordination bottleneck: before denoising begins, prompt-conditioned attention may allocate different concepts to strongly overlapping spatial support, which can keep their attention coupled as denoising proceeds. This observation motivates treating compositional generation as a boundary-condition problem rather than repeatedly controlling the evolving trajectory. To this end, we propose Rectify-then-Diffuse (RTD), a training-free framework that rectifies the initial allocation once before standard denoising. Firstly, we propose Soft-Overlap Disentanglement (SOD), which converts normalized overlap between pilot concept maps into a differentiable and layout-agnostic separation objective. Secondly, we introduce Isotropic Gradient Rectification (IGR), which normalizes the SOD gradient and applies a bounded latent displacement with a consistent scale across prompts and initializations. Extensive experiments show that RTD achieves state-of-the-art compositional fidelity and robust gains. On the AE-Bench object pair subset, RTD improves BLIP-VQA by 45.8% and ImageReward by 19.6% over CO3 while running 2.3

\times

faster. Code will be released at https://github.com/Z-yiwei/rectify-then-diffuse

Abstract

Compositional T2I Generation

Explore similar work

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

ORCA: Hunting Compositional Failures in Text-to-Image Diffusion

Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time