Organizations: Ritsumeikan University, Ibaraki, Osaka 567-8570, Japan · The Institute of Statistical Mathematics, Tachikawa, Tokyo 190-8562, Japan · Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan · Kyoto University, Sakyo-ku, Kyoto 606-8501, Japan
Text-to-image generation models do not explicitly receive the syntactic structure of the prompt, i.e., which word modifies which and where each phrase attaches. Consequently, they often struggle to bind attributes to the correct objects, to depict spatial relations and counts correctly, and to render syntactically ambiguous prompts in the intended interpretation. Existing syntax-based control methods mainly address attribute binding and have limited ability to control spatial relations and counts or to specify the interpretation of syntactically ambiguous prompts. They rely on phrase-structure or dependency grammars and require additional rules or constraints on the tree structure to extract modification relations between words or to specify a syntactic interpretation. We propose CCG Diffusion, a training-free method that controls image generation using Combinatory Categorial Grammar (CCG). In CCG, the category of each word carries rich syntactic information: it specifies what the word combines with and what phrase it forms. Modification relations between words can therefore be read directly from a CCG parse. Moreover, prepositional phrase (PP) attachment ambiguity, a representative syntactic ambiguity, can be resolved simply by fixing the preposition's category. CCG Diffusion applies the same two CCG-based mechanisms, prompt rewriting and attention guidance, both to controlling attribute binding, spatial relations, and counts and to resolving PP ambiguity. On two architecturally different backbones, Stable Diffusion and Sana, CCG Diffusion achieves the highest scores among the compared methods in attribute binding and 2D spatial relations and significantly outperforms all of them in PP ambiguity resolution.
Figures & tables
Fig. 1: The two target problems addressed in this paper. (a) Compositional text-to-image generation: generating an image that matches a prompt describing multiple objects and their attributes, relations, and counts. (b) Syntactic ambiguity resolution: generating an image that follows the intended interpretation of a prompt that admits multiple syntactic structures, e.g., whether a PP modifies a noun phrase (NP attachment) or a verb phrase (VP attachment). The proposed CCG Diffusion can address both of these problems.
Type
Targets
Proposition
Existence (exist)
N∈E
∃x.N(x)
Correspondence (corr)
(M,N)∈C
∀x.M(x)↔N(x)
Leakage (leak)
(M,N)∈K
∀x.M(x)→¬N(x)
Possession (poss)
(S,O)∈P
∀x.O(x)→S(x)
TABLE I: Defined proposition types in Predicated Diffusion.
Fig. 2: CCG derivation of an example sentence: a blue banana and a green vase . The category of each word is shown below it. Each line indicates that two adjacent phrases are combined by the combinatory rule shown at its right end ( > : forward application; Φ : coordination), yielding the category below the line. Blue and green mark the derivations of the two noun phrases.
Fig. 3: Overview of CCG Diffusion. A CCG parser parses the prompt into a derivation D ; for PP ambiguity resolution, it parses the prompt under the specified NP or VP attachment. The syntactic information read from D is conveyed to the image generation model through two pathways: (a) the input prompt to the text encoder and (b) attention guidance.
Fig. 4: CCG derivations of an example sentence: the boy looks at the girl with a telescope under the two interpretations. The derivations are obtained by parsing the prompt with the CCG parser while fixing the supertag of with (boxed) to (a) (NP\NP)/NP for NP attachment and (b) ((S\NP)\(S\NP))/NP for VP attachment.
Attribute binding
Spatial
Numeracy
Non-spatial
Complex
Method
Color
Shape
Texture
2D
3D
(UniDet [ 72 ] )
(CLIPScore [ 73 ] )
(3-in-1)
Stable Diffusion
Vanilla
0.3794
0.3595
0.4169
0.1148
0.2957
0.4464
0.3108
0.2963
Composable Diffusion [ 53 ]
0.4198
0.3560
0.4301
0.1115
0.3056
0.4364
0.3085
0.3038
StructureDiffusion [ 28 ]
0.3587
0.3598
0.3987
0.1082
0.3039
0.4496
0.3093
0.3022
Attend-and-Excite [ 9 ]
0.5761
0.4802
0.6106
0.1043
0.3025
0.5191
0.3088
0.3279
TABLE II: Performance on T2I-CompBench++. Higher is better in all categories. Bold: significantly higher than all other rows of the same backbone. † : CCG Diffusion significantly higher than Predicated Diffusion (paired permutation tests over the 300 prompts, Holm-corrected over the eight subcategories of each backbone, p<0.05 ).
Stable Diffusion
Sana
Removed component
Color
Shape
Texture
Color
Shape
Texture
Selective Guidance Boost
− 0.032 ***
− 0.041 ***
− 0.048 ***
− 0.012 **
− 0.010 *
− 0.031 ***
Positional Prefix
− 0.021 **
− 0.035 ***
− 0.016 ***
− 0.065 ***
− 0.044 ***
− 0.022 ***
Region Assignment Loss
− 0.030 ***
− 0.014 **
− 0.011 **
− 0.006 n.s.
− 0.002 n.s.
− 0.003 n.s.
TABLE III: Score drop from removing each component on attribute binding in T2I-CompBench++. Score with the listed component disabled minus the score with all three components enabled (two-sided paired permutation test over the 300 prompts, Holm-corrected over the nine comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Fig. 5: Qualitative comparison on Stable Diffusion (top) and Sana (bottom) for the Color, Shape, Texture, 2D-spatial, and Numeracy categories.
Removed objective (category)
Stable Diffusion
Sana
Spatial Relation Loss (2D)
− 0.177 ***
− 0.122 ***
Count Loss (numeracy)
− 0.013 **
− 0.005 n.s.
TABLE IV: Score drop from removing the Spatial Relation Loss and the Count Loss in T2I-CompBench++. T2I-CompBench++ score with each objective removed minus the score of CCG Diffusion, evaluated on the category that the objective targets (two-sided paired permutation test over the 300 prompts, Holm-corrected over the two comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Fig. 6: Examples of generated images for the TAB PP-attachment category on Stable Diffusion and Sana.
TABLE V: TAB PP comparison on Stable Diffusion and Sana. Bold: significantly higher than all other rows of the same backbone under the same VQA model. † : CCG Diffusion significantly higher than Predicated Diffusion (paired permutation test over the 74 prompts, Holm-corrected over the two VQA models per backbone, p<0.05 ). Cohen’s κ : agreement between the judgments of the two VQA models over the 740 trials of each row.
Removed component
Stable Diffusion
Sana
Disambiguating Paraphrase
− 4.1 n.s.
− 18.0 ***
Region Assignment Loss
− 2.0 n.s.
− 0.9 n.s.
Both
− 7.4 **
− 17.7 ***
TABLE VI: Yes-rate drop from removing each component of CCG Diffusion on TAB PP. Yes rate (%, judged by GPT-6 Astra) with each component or both disabled minus the Yes rate with both components enabled (two-sided paired permutation test over the 74 prompts, Holm-corrected over the three comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Figure 21Figure 22Figure 23Figure 24Figure 25
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Stable Diffusion
Sana
Resolution
512 × 512
Denoising steps
50 (PNDM)
20
Classifier-free guidance
7.5
4.5
Guided steps
first 25
all 20
Extra updates at step 0
5
5
Attention maps
5 layers (16 × 16)
first 8 blocks
Appendix
TABLE VII: Generation settings and hyperparameters.
Method
Stable Diffusion
Sana
Vanilla
1.77
0.80
Predicated Diffusion
3.77
1.89
CCG Diffusion (ours)
3.90
1.91
Appendix
TABLE VIII: Average generation time per image (s) on T2I-CompBench++.
Removed component
Stable Diffusion
Sana
Selective Guidance Boost
Positional Prefix
Region Assignment Loss
Color
Shape
Texture
Color
Shape
Texture
✓
✓
✓
− 0.070 ***
− 0.075 ***
− 0.068 ***
− 0.083 ***
− 0.069 ***
− 0.048 ***
✓
✓
− 0.058 ***
− 0.069 ***
− 0.063 ***
− 0.079 ***
− 0.068 ***
− 0.047 ***
✓
✓
− 0.037 ***
− 0.055 ***
− 0.054 ***
− 0.011 **
− 0.010 **
− 0.032 ***
✓
✓
− 0.046 ***
− 0.041 ***
− 0.017 ***
− 0.078 ***
− 0.047 ***
− 0.026 ***
Appendix
TABLE IX: Score drop from removing two or more components on attribute binding in T2I-CompBench++. ✓ marks the disabled components. Score of each configuration minus the score with all three components enabled (two-sided paired permutation test over the 300 prompts, Holm-corrected over the 12 comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 .
Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.
Modern text-to-image models produce high-fidelity images but still struggle with compositional prompts that require instance identity, attribute ownership, counting, spatial ordering, and role-sensitive relations. We introduce Panoptic Scene Program Diffusion Transformer (PSP-DiT), a diffusion-transformer architecture that treats a panoptic scene program as a first-class latent variable rather than an external control signal or post-hoc parse. PSP-DiT jointly denoises image latents and scene-program latents through coupled transformer streams, while panoptic grounding and cycle-consistency objectives tie object instances, attributes, relations, and counts to visual support in the generated image. Under matched training and inference settings, PSP-DiT improves over a strong flat-text baseline across GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster, with the largest gains on counting, attribute binding, role-sensitive relations, and long structured prompts. The method preserves image quality, adds modest inference overhead, and remains robust to imperfect scene programs.
Text-to-image diffusion models can generate individual concepts well, but they often omit or merge concepts incorrectly with multiple concepts. We trace these failures to an early coordination bottleneck: before denoising begins, prompt-conditioned attention may allocate different concepts to strongly overlapping spatial support, which can keep their attention coupled as denoising proceeds. This observation motivates treating compositional generation as a boundary-condition problem rather than repeatedly controlling the evolving trajectory. To this end, we propose Rectify-then-Diffuse (RTD), a training-free framework that rectifies the initial allocation once before standard denoising. Firstly, we propose Soft-Overlap Disentanglement (SOD), which converts normalized overlap between pilot concept maps into a differentiable and layout-agnostic separation objective. Secondly, we introduce Isotropic Gradient Rectification (IGR), which normalizes the SOD gradient and applies a bounded latent displacement with a consistent scale across prompts and initializations. Extensive experiments show that RTD achieves state-of-the-art compositional fidelity and robust gains. On the AE-Bench object pair subset, RTD improves BLIP-VQA by 45.8% and ImageReward by 19.6% over CO3 while running 2.3× faster. Code will be released at https://github.com/Z-yiwei/rectify-then-diffuse
Ning Zhu, An Chen, Mengfei Zhao +4
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China
Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same prompts. We formalize this as the understanding-generation gap and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a prompt, the LLM first produces a coarse visual draft composed of discrete vision tokens. It then performs a self-critique by evaluating the draft for prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a diffusion model is conditioned jointly on the prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions, hallucinations, and relational errors. Experiments show that UniReasoner improves compositional alignment and semantic faithfulness under the same diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM reasoning to close the understanding-generation gap.