Organizations: Ritsumeikan University, Ibaraki, Osaka 567-8570, Japan · The Institute of Statistical Mathematics, Tachikawa, Tokyo 190-8562, Japan · Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan · Kyoto University, Sakyo-ku, Kyoto 606-8501, Japan
Text-to-image generation models do not explicitly receive the syntactic structure of the prompt, i.e., which word modifies which and where each phrase attaches. Consequently, they often struggle to bind attributes to the correct objects, to depict spatial relations and counts correctly, and to render syntactically ambiguous prompts in the intended interpretation. Existing syntax-based control methods mainly address attribute binding and have limited ability to control spatial relations and counts or to specify the interpretation of syntactically ambiguous prompts. They rely on phrase-structure or dependency grammars and require additional rules or constraints on the tree structure to extract modification relations between words or to specify a syntactic interpretation. We propose CCG Diffusion, a training-free method that controls image generation using Combinatory Categorial Grammar (CCG). In CCG, the category of each word carries rich syntactic information: it specifies what the word combines with and what phrase it forms. Modification relations between words can therefore be read directly from a CCG parse. Moreover, prepositional phrase (PP) attachment ambiguity, a representative syntactic ambiguity, can be resolved simply by fixing the preposition's category. CCG Diffusion applies the same two CCG-based mechanisms, prompt rewriting and attention guidance, both to controlling attribute binding, spatial relations, and counts and to resolving PP ambiguity. On two architecturally different backbones, Stable Diffusion and Sana, CCG Diffusion achieves the highest scores among the compared methods in attribute binding and 2D spatial relations and significantly outperforms all of them in PP ambiguity resolution.
Figures & tables
Fig. 1: The two target problems addressed in this paper. (a) Compositional text-to-image generation: generating an image that matches a prompt describing multiple objects and their attributes, relations, and counts. (b) Syntactic ambiguity resolution: generating an image that follows the intended interpretation of a prompt that admits multiple syntactic structures, e.g., whether a PP modifies a noun phrase (NP attachment) or a verb phrase (VP attachment). The proposed CCG Diffusion can address both of these problems.
Type
Targets
Proposition
Existence (exist)
N∈E
∃x.N(x)
Correspondence (corr)
(M,N)∈C
∀x.M(x)↔N(x)
Leakage (leak)
(M,N)∈K
∀x.M(x)→¬N(x)
Possession (poss)
(S,O)∈P
∀x.O(x)→S(x)
TABLE I: Defined proposition types in Predicated Diffusion.
Fig. 2: CCG derivation of an example sentence: a blue banana and a green vase . The category of each word is shown below it. Each line indicates that two adjacent phrases are combined by the combinatory rule shown at its right end ( > : forward application; Φ : coordination), yielding the category below the line. Blue and green mark the derivations of the two noun phrases.
Fig. 3: Overview of CCG Diffusion. A CCG parser parses the prompt into a derivation D ; for PP ambiguity resolution, it parses the prompt under the specified NP or VP attachment. The syntactic information read from D is conveyed to the image generation model through two pathways: (a) the input prompt to the text encoder and (b) attention guidance.
Fig. 4: CCG derivations of an example sentence: the boy looks at the girl with a telescope under the two interpretations. The derivations are obtained by parsing the prompt with the CCG parser while fixing the supertag of with (boxed) to (a) (NP\NP)/NP for NP attachment and (b) ((S\NP)\(S\NP))/NP for VP attachment.
Attribute binding
Spatial
Numeracy
Non-spatial
Complex
Method
Color
Shape
Texture
2D
3D
(UniDet [ 72 ] )
(CLIPScore [ 73 ] )
(3-in-1)
Stable Diffusion
Vanilla
0.3794
0.3595
0.4169
0.1148
0.2957
0.4464
0.3108
0.2963
Composable Diffusion [ 53 ]
0.4198
0.3560
0.4301
0.1115
0.3056
0.4364
0.3085
0.3038
StructureDiffusion [ 28 ]
0.3587
0.3598
0.3987
0.1082
0.3039
0.4496
0.3093
0.3022
Attend-and-Excite [ 9 ]
0.5761
0.4802
0.6106
0.1043
0.3025
0.5191
0.3088
0.3279
TABLE II: Performance on T2I-CompBench++. Higher is better in all categories. Bold: significantly higher than all other rows of the same backbone. † : CCG Diffusion significantly higher than Predicated Diffusion (paired permutation tests over the 300 prompts, Holm-corrected over the eight subcategories of each backbone, p<0.05 ).
Stable Diffusion
Sana
Removed component
Color
Shape
Texture
Color
Shape
Texture
Selective Guidance Boost
− 0.032 ***
− 0.041 ***
− 0.048 ***
− 0.012 **
− 0.010 *
− 0.031 ***
Positional Prefix
− 0.021 **
− 0.035 ***
− 0.016 ***
− 0.065 ***
− 0.044 ***
− 0.022 ***
Region Assignment Loss
− 0.030 ***
− 0.014 **
− 0.011 **
− 0.006 n.s.
− 0.002 n.s.
− 0.003 n.s.
TABLE III: Score drop from removing each component on attribute binding in T2I-CompBench++. Score with the listed component disabled minus the score with all three components enabled (two-sided paired permutation test over the 300 prompts, Holm-corrected over the nine comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Fig. 5: Qualitative comparison on Stable Diffusion (top) and Sana (bottom) for the Color, Shape, Texture, 2D-spatial, and Numeracy categories.
Removed objective (category)
Stable Diffusion
Sana
Spatial Relation Loss (2D)
− 0.177 ***
− 0.122 ***
Count Loss (numeracy)
− 0.013 **
− 0.005 n.s.
TABLE IV: Score drop from removing the Spatial Relation Loss and the Count Loss in T2I-CompBench++. T2I-CompBench++ score with each objective removed minus the score of CCG Diffusion, evaluated on the category that the objective targets (two-sided paired permutation test over the 300 prompts, Holm-corrected over the two comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Fig. 6: Examples of generated images for the TAB PP-attachment category on Stable Diffusion and Sana.
TABLE V: TAB PP comparison on Stable Diffusion and Sana. Bold: significantly higher than all other rows of the same backbone under the same VQA model. † : CCG Diffusion significantly higher than Predicated Diffusion (paired permutation test over the 74 prompts, Holm-corrected over the two VQA models per backbone, p<0.05 ). Cohen’s κ : agreement between the judgments of the two VQA models over the 740 trials of each row.
Removed component
Stable Diffusion
Sana
Disambiguating Paraphrase
− 4.1 n.s.
− 18.0 ***
Region Assignment Loss
− 2.0 n.s.
− 0.9 n.s.
Both
− 7.4 **
− 17.7 ***
TABLE VI: Yes-rate drop from removing each component of CCG Diffusion on TAB PP. Yes rate (%, judged by GPT-6 Astra) with each component or both disabled minus the Yes rate with both components enabled (two-sided paired permutation test over the 74 prompts, Holm-corrected over the three comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 , n.s. not significant.
Figure 21Figure 22Figure 23Figure 24Figure 25
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Stable Diffusion
Sana
Resolution
512 × 512
Denoising steps
50 (PNDM)
20
Classifier-free guidance
7.5
4.5
Guided steps
first 25
all 20
Extra updates at step 0
5
5
Attention maps
5 layers (16 × 16)
first 8 blocks
Appendix
TABLE VII: Generation settings and hyperparameters.
Method
Stable Diffusion
Sana
Vanilla
1.77
0.80
Predicated Diffusion
3.77
1.89
CCG Diffusion (ours)
3.90
1.91
Appendix
TABLE VIII: Average generation time per image (s) on T2I-CompBench++.
Removed component
Stable Diffusion
Sana
Selective Guidance Boost
Positional Prefix
Region Assignment Loss
Color
Shape
Texture
Color
Shape
Texture
✓
✓
✓
− 0.070 ***
− 0.075 ***
− 0.068 ***
− 0.083 ***
− 0.069 ***
− 0.048 ***
✓
✓
− 0.058 ***
− 0.069 ***
− 0.063 ***
− 0.079 ***
− 0.068 ***
− 0.047 ***
✓
✓
− 0.037 ***
− 0.055 ***
− 0.054 ***
− 0.011 **
− 0.010 **
− 0.032 ***
✓
✓
− 0.046 ***
− 0.041 ***
− 0.017 ***
− 0.078 ***
− 0.047 ***
− 0.026 ***
Appendix
TABLE IX: Score drop from removing two or more components on attribute binding in T2I-CompBench++. ✓ marks the disabled components. Score of each configuration minus the score with all three components enabled (two-sided paired permutation test over the 300 prompts, Holm-corrected over the 12 comparisons per backbone). * p<0.05 , ** p<0.01 , *** p<0.001 .
Compositional text-to-image generation creates images that faithfully combine multiple objects, attributes, relationships, and styles specified in a prompt.
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China