cs.CVSep 24, 2026
SavePanoptic Scene Program Diffusion Transformer
Organizations: William & Mary, Williamsburg, VA, USA
Abstract
Modern text-to-image models produce high-fidelity images but still struggle with compositional prompts that require instance identity, attribute ownership, counting, spatial ordering, and role-sensitive relations. We introduce Panoptic Scene Program Diffusion Transformer (PSP-DiT), a diffusion-transformer architecture that treats a panoptic scene program as a first-class latent variable rather than an external control signal or post-hoc parse. PSP-DiT jointly denoises image latents and scene-program latents through coupled transformer streams, while panoptic grounding and cycle-consistency objectives tie object instances, attributes, relations, and counts to visual support in the generated image. Under matched training and inference settings, PSP-DiT improves over a strong flat-text baseline across GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster, with the largest gains on counting, attribute binding, role-sensitive relations, and long structured prompts. The method preserves image quality, adds modest inference overhead, and remains robust to imperfect scene programs.
Figures & tables
Figure 1: Overview of PSP-DiT. PSP-DiT augments the image latent with a jointly denoised panoptic scene-program latent that stores object instances, attributes, relations, counts, and scene context. Coupled image/program transformer streams exchange information through bidirectional cross-attention, while grounding and cycle-consistency heads tie scene variables to visual ownership and recoverable structure.
Figure 2: Qualitative compositional stress cases . We compare FLUX.1-dev, SD3.5-Large, and PSP-DiT using the same prompts, random seeds, and matched inference budget. Rows test: (1) counting and attribute ownership, with three mugs, a spoon only in the blue mug, and an apple in front of the red mug; (2) attribute binding, with a green apple on a white plate, a red banana on a black plate, and a spoon between them; (3) role-sensitive spatial reasoning, with a dog behind a cat, the cat carrying a red ball, and a frisbee to the right; and (4) long structured composition, with classroom roles, notebook ownership, a yellow-shirt student, and exactly two pencils. The baselines often produce visually plausible images while losing instance-level constraints. PSP-DiT more consistently preserves instance identity, attribute ownership, ordered relations, and visual support, qualitatively corroborating the trends in Table 1 (a,c). Full prompts are provided in Appendix A.12 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Study component | Protocol |
| Prompt source | PartiPrompts held-out subset |
| Number of prompts | 200 |
| Raters per prompt | 3 |
| Total judgments per comparison | 600 |
| Comparison type | Blinded pairwise forced choice |
| Displayed instruction | Better prompt satisfaction and visual quality |
Table 3: Human preference protocol. We use a blinded forced-choice study on held-out PartiPrompts prompts to evaluate prompt satisfaction and visual quality under matched inference settings.
| Model | GenEval 2 | PSG-Score | DetailMaster Avg. |
| FLUX.1-dev | 29.5 | 0.62 | 63.6 |
| SD3.5-Large | 19.0 | 0.60 | 59.6 |
Table 4: Appendix positioning tables using literature-reported public numbers. Top: overlapping public open-baseline subset with matched coverage across GenEval 2, PSG-Score, and DetailMaster. Bottom: frontier public-reference systems on SANEval (reported SANEval-Simple averaged score).
| Setup | Latency (s/img) | Peak VRAM (GB) |
| FLUX.1-dev RTX A6000, 1024 1024, 50 steps | 49.7 | 32.8 |
Table 5: Published external systems anchor for FLUX.1-dev . This reference is provided only for context; unlike our H200 measurements, it is not directly comparable because latency and VRAM depend strongly on hardware and software stack.
| Skill | Contributing sub-benchmark(s) | Prompt-count source |
| Counting | SANEval-Simple: Numeracy | released SANEval-Simple numeracy split |
| Attr. Binding | SANEval-Simple: Attribute Binding | released SANEval-Simple attribute-binding split |
| Spatial Rel. | SANEval-Simple: Spatial Relations | released SANEval-Simple spatial-relations split |
| Role Rel. | GenEval 2 (Rewritten): transitive-verb / role-sensitive subset | released GenEval 2 rewritten role-sensitive subset |
| Long Prompts | DetailMaster: full benchmark | public benchmark total: 4,116 prompts |
Table 6: Skill-to-benchmark mapping used for Table 1 . Each skill score is computed from the listed benchmark-native subset and linearly rescaled to using the benchmark’s documented range.
| Row | Skill stress | Full prompt |
| 1 | Counting; attribute ownership; left–center–right ordering | A photorealistic tabletop scene, viewed from the front, with exactly three ceramic mugs in a straight row: a red mug on the left, a blue mug in the center, and a yellow mug on the right. A silver spoon is inside only the blue mug, and a small green apple sits directly in front of the red mug. |
| 2 | Attribute binding; object–support assignment | A photorealistic close-up of a green apple on a white plate to the left of a red banana on a black plate, with a wooden spoon placed between the two plates. |
| 3 | Role-sensitive relation; spatial ordering; object ownership | A photorealistic park scene where a brown dog is chasing a white cat across a path; the dog is visibly behind the cat, the cat is carrying a red ball in its mouth, and a blue frisbee lies on the grass to the right of both animals. |
| 4 | Long structured prompt; roles; counting; attribute ownership | A photorealistic classroom: a teacher in a gray sweater points at a square on the blackboard while a student in a red shirt holds a blue notebook; another student in a yellow shirt sits behind the desk, and exactly two pencils lie on the desk. |
Table 7: Full prompts used for the main qualitative stress cases in Figure 2 . The same prompt, random seed, and matched inference budget are used across FLUX.1-dev, SD3.5-Large, and PSP-DiT for each row.
Figure 3: Additional qualitative compositional stress cases. These examples extend Figure 2 to additional prompts covering counting, attribute binding, spatial relations, object interaction, and long structured descriptions.
Figure 4: Additional qualitative compositional stress cases (continued).
Figure 5: Additional qualitative compositional stress cases (continued).
Figure 6: Additional qualitative compositional stress cases (continued).
Figure 7: Additional qualitative compositional stress cases (continued).
Figure 8: Additional qualitative compositional stress cases (continued).
More in this topic
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 463 papers