Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a 63.4% relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of −9.58. Meanwhile, our method offers a 2.3× to 12× real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.
Figures & tables
Figure 1: Scatter plots of compositional generation error vs FID on 3 datasets (3 input components). Our method lies on the Pareto front of all results (see Appendix for full scatter plots) while achieving the lowest or joint lowest error among the baselines.
Figure 2: Compositional text-to-image results with captions (zooming recommended). Top: single-prompt baseline. Bottom: composed multi-prompt (ours). Our method enables the composition of multiple conditions, conferring an advantage over the single-prompt baseline.
Figure 3: Concept negation with text-to-image (left baseline, right ours): Our method allows more precise control over the outputs of an existing pre-trained model (aMUSEd [ patil2024amused ] ). The baseline handles poorly the negation/removal of characteristics which commonly co-occur with the subject of the image (e.g. a king with no crown). Each image is selected from three runs as the most representative of the prompt (ours and baseline).
Figure 4: Conceptual product space: Example of composing two concept spaces using our framework: {"a cat","a dog","an apple","a cherry"} ×\lx@scalerel@objAND {"a red object","an orange object","a yellow object","a green object","a blue object","an indigo object","a violet object"}.
Figure 5: Overview of our approach. Left : We start with a generic “empty” state s0 or the preceding state st and a set of input concepts c1,c2,... . These are used to compute the unconditional distribution over st+1 and the conditional distributions given each of c1,c2,... . These are composed , optionally incorporating concept weights , to produce an accurate estimate of the distribution conditioned on all inputs. The process is repeated until a “full” state sT is obtained, e.g. a grid of VQ-VAE latents. Right : Our general formulation encompasses any additive discrete generative process, i.e. where the representation of st is fully determined by st+1 (masked and autoregressive generation shown as examples).
Figure 6: Effect of varying the wsmile concept weight from −3.0 to 3.0 while keeping wmale=wno_glasses=3.0 .
Figure 7: Compositional out-of-distribution generation: Positional CLEVR training images contain no more than 5 objects per image, but our compositional method allows 6 or more objects to appear in the same image via compositional sampling.
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
62.72
±1.37
57.41
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
98.96
±0.29
51.37
99.96
±0.04
23.29
100.00
±0.00
19.01
LACE [ nie2021controllable ]
99.30
±0.24
50.92
100.00
±0.00
22.83
100.00
±0.00
19.62
GLIDE [ nichol2021glide ]
99.14
±0.26
61.68
99.94
±0.06
38.26
100.00
±0.00
37.18
EBM [ du2020compositional ]
29.46
±1.29
78.63
71.78
±1.27
65.45
92.66
±0.74
58.33
Table 1: Quantitative results (error rate and FID score) on the Positional CLEVR dataset
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
32.29
±1.32
20.55
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
79.82
±1.14
22.29
98.34
±0.36
30.58
99.84
±0.11
31.30
LACE [ nie2021controllable ]
98.90
±0.30
40.54
99.90
±0.09
40.61
99.96
±0.04
40.60
GLIDE [ nichol2021glide ]
53.80
±1.41
17.61
91.14
±0.80
28.56
98.64
±0.33
40.02
EBM [ du2020compositional ]
21.86
±1.17
44.41
75.84
±1.21
55.89
95.74
±0.57
58.66
Table 2: Quantitative results (error rate and FID score) on the Relational CLEVR dataset
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
8.94
±0.81
10.75
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
41.10
±1.39
18.04
69.32
±1.30
18.06
83.04
±1.06
18.06
LACE [ nie2021controllable ]
2.40
±0.43
28.21
4.34
±0.58
36.23
19.12
±1.11
34.64
GLIDE [ nichol2021glide ]
1.34
±0.33
20.30
51.32
±1.41
22.69
72.76
±1.26
21.98
EBM [ du2020compositional ]
1.26
±0.32
89.95
6.90
±0.72
99.64
69.99
±1.30
335.70
Table 3: Quantitative results (error rate and FID score) on the FFHQ dataset
Method
Parameters
Sample Time
Sample Time
Per Image
Per Batch of 25
EBM
33 mil
5.99s ± 0.17
108.57s ± 0.93
Composed GLIDE
385 mil
4.92s ±0.17
73.92s ± 0.70
Ours
108 mil
2.11s ± 0.39
9.08s ± 0.39
Table 4: Parameter counts and sample times for 3 input components