Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a 63.4% relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of −9.58. Meanwhile, our method offers a 2.3× to 12× real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.
Figures & tables
Figure 1: Scatter plots of compositional generation error vs FID on 3 datasets (3 input components). Our method lies on the Pareto front of all results (see Appendix for full scatter plots) while achieving the lowest or joint lowest error among the baselines.
Figure 2: Compositional text-to-image results with captions (zooming recommended). Top: single-prompt baseline. Bottom: composed multi-prompt (ours). Our method enables the composition of multiple conditions, conferring an advantage over the single-prompt baseline.
Figure 3: Concept negation with text-to-image (left baseline, right ours): Our method allows more precise control over the outputs of an existing pre-trained model (aMUSEd [ patil2024amused ] ). The baseline handles poorly the negation/removal of characteristics which commonly co-occur with the subject of the image (e.g. a king with no crown). Each image is selected from three runs as the most representative of the prompt (ours and baseline).
Figure 4: Conceptual product space: Example of composing two concept spaces using our framework: {"a cat","a dog","an apple","a cherry"} ×\lx@scalerel@objAND {"a red object","an orange object","a yellow object","a green object","a blue object","an indigo object","a violet object"}.
Figure 5: Overview of our approach. Left : We start with a generic “empty” state s0 or the preceding state st and a set of input concepts c1,c2,... . These are used to compute the unconditional distribution over st+1 and the conditional distributions given each of c1,c2,... . These are composed , optionally incorporating concept weights , to produce an accurate estimate of the distribution conditioned on all inputs. The process is repeated until a “full” state sT is obtained, e.g. a grid of VQ-VAE latents. Right : Our general formulation encompasses any additive discrete generative process, i.e. where the representation of st is fully determined by st+1 (masked and autoregressive generation shown as examples).
Figure 6: Effect of varying the wsmile concept weight from −3.0 to 3.0 while keeping wmale=wno_glasses=3.0 .
Figure 7: Compositional out-of-distribution generation: Positional CLEVR training images contain no more than 5 objects per image, but our compositional method allows 6 or more objects to appear in the same image via compositional sampling.
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
62.72
±1.37
57.41
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
98.96
±0.29
51.37
99.96
±0.04
23.29
100.00
±0.00
19.01
LACE [ nie2021controllable ]
99.30
±0.24
50.92
100.00
±0.00
22.83
100.00
±0.00
19.62
GLIDE [ nichol2021glide ]
99.14
±0.26
61.68
99.94
±0.06
38.26
100.00
±0.00
37.18
EBM [ du2020compositional ]
29.46
±1.29
78.63
71.78
±1.27
65.45
92.66
±0.74
58.33
Table 1: Quantitative results (error rate and FID score) on the Positional CLEVR dataset
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
32.29
±1.32
20.55
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
79.82
±1.14
22.29
98.34
±0.36
30.58
99.84
±0.11
31.30
LACE [ nie2021controllable ]
98.90
±0.30
40.54
99.90
±0.09
40.61
99.96
±0.04
40.60
GLIDE [ nichol2021glide ]
53.80
±1.41
17.61
91.14
±0.80
28.56
98.64
±0.33
40.02
EBM [ du2020compositional ]
21.86
±1.17
44.41
75.84
±1.21
55.89
95.74
±0.57
58.66
Table 2: Quantitative results (error rate and FID score) on the Relational CLEVR dataset
Method
1 Component
2 Components
3 Components
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
Err (%) ↓
FID ↓
StyleGAN2-ADA [ karras2020training ]
8.94
±0.81
10.75
-
-
-
-
StyleGAN2 [ karras2020analyzing ]
41.10
±1.39
18.04
69.32
±1.30
18.06
83.04
±1.06
18.06
LACE [ nie2021controllable ]
2.40
±0.43
28.21
4.34
±0.58
36.23
19.12
±1.11
34.64
GLIDE [ nichol2021glide ]
1.34
±0.33
20.30
51.32
±1.41
22.69
72.76
±1.26
21.98
EBM [ du2020compositional ]
1.26
±0.32
89.95
6.90
±0.72
99.64
69.99
±1.30
335.70
Table 3: Quantitative results (error rate and FID score) on the FFHQ dataset
Method
Parameters
Sample Time
Sample Time
Per Image
Per Batch of 25
EBM
33 mil
5.99s ± 0.17
108.57s ± 0.93
Composed GLIDE
385 mil
4.92s ±0.17
73.92s ± 0.70
Ours
108 mil
2.11s ± 0.39
9.08s ± 0.39
Table 4: Parameter counts and sample times for 3 input components
Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models. Yet, not all mechanisms that enable or inhibit it are fully understood. In this work, we conduct a systematic study of which design choices critically determine compositional generalization in image and video generation. By isolating independent design axes, we identify two key factors strongly associated with compositional success: (i) whether the training objective operates on a discrete or continuous distribution, and (ii) the completeness of conditioning information about constituent factors during training. We also show that relaxing the discrete loss with an auxiliary continuous latent objective can partially recover compositional performance in discrete models like MaskGIT. Our findings, corroborated by diverse compositional tasks and preliminary evidence in world models and LLMs, motivate a shift toward continuous objectives for compositional generalization.
Karim Farid, Rajat Sahay, Yumna Ali Alnaggar +4
University of Freiburg · Bosch Center for Artificial Intelligence · Inria, École Normale Supérieure, CNRS, PSL Research University
Current text-to-image models struggle to provide precise camera control using natural language alone. In this work, we present a framework for precise camera control with global scene understanding in text-to-image generation by learning parametric camera tokens. We fine-tune image generation models for viewpoint-conditioned text-to-image generation on a curated dataset that combines 3D-rendered images for geometric supervision and photorealistic augmentations for appearance and background diversity. Qualitative and quantitative experiments demonstrate that our method achieves state-of-the-art accuracy while preserving image quality and prompt fidelity. Unlike prior methods that overfit to object-specific appearance correlations, our viewpoint tokens learn factorized geometric representations that transfer to unseen object categories. Our work shows that text-vision latent spaces can be endowed with explicit 3D camera structure, offering a pathway toward geometrically-aware prompts for text-to-image generation. Project page: https://randdl.github.io/viewtoken_control/
Discrete autoregressive (AR) text-to-image (T2I) models pair a VQ tokenizer with an AR policy, and current post-training pipelines optimize only the policy while keeping the VQ decoder frozen. Recent diffusion T2I work, exemplified by REPA-E, has shown that the VAE itself constitutes a key alignment bottleneck, yet no analogous investigation exists for discrete AR models. We show that policy-only optimization induces Latent Covariate Shift: as the policy evolves, the resulting token distribution diverges from the ground-truth distribution on which the decoder was trained, such that reward scores improve while decoded image quality degrades. To address this mismatch, we propose RankE, the first end-to-end post-training framework for discrete T2I generation. Rather than optimizing the policy against a fixed decoder, RankE co-evolves both components through alternating optimization: each module maximizes a ranking-based alignment objective while being regularized by a stability-preserving anchor suited to its parameter space. This co-evolution breaks the fidelity--alignment trade-off that plagues frozen-decoder approaches: on LlamaGen-XL (775M), standard RL improves CLIP but degrades FID, whereas RankE improves both simultaneously (FID 15.21, CLIP 33.76 on MS-COCO 30K). Consistent gains on Janus-Pro (1B) confirm that decoder co-evolution reliably converts reward optimization into pixel-space quality improvements.
Siyong Jian, Siyuan Li, Luyuan Zhang +5
1Westlake University · 2Zhejiang University · 3Tsinghua University +2