ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images
Organizations: Canva · Queen Mary University of London · Universidade Estadual de Maring´a
Abstract
Graphic designs are ordered compositions of RGBA layers whose separation enables selection, movement, and reuse. Existing layered models represent each layer with a full-canvas or region-sized spatial latent grid, so representation size grows with the cumulative spatial extent of the layers. We present ProtoDiffusion, a latent-diffusion framework built on ProtoVAE, a multi-layer variational autoencoder that encodes a design into one shared spatial prototype and one candidate vector per layer. Each candidate conditions a shared decoder to reconstruct its corresponding RGBA layer. For layers, the representation contains tokens before backbone-specific packing, compared with for full-grid-per-layer representations. ProtoDiffusion jointly denoises the prototype and a fixed set of candidate slots, conditioned on an image or text, with bounding boxes as optional layout guidance. The model leaves unneeded slots empty and determines the number of layers and, in absence of boxes, their layout, requiring neither as input. The same architecture supports both image-to-design and text-to-design. ProtoVAE achieves high-fidelity reconstruction from its compact shared latent representation, reaching 40.43 dB PSNR on layer crops. On the LayerD evaluation split, image-to-design achieves RGB L1 scores of 0.040 with bounding boxes and 0.057 without, with box-guided alpha soft IoU (0.850) comparable to the strongest fine-tuned baseline (0.871). On merged images, text-to-design reaches 66.41 FID versus 86.84 for FLUX.2 [klein], achieves a 0.860 text-accuracy score, and runs faster than the same-backbone generation-then-decomposition cascade.
Figures & tables
| Full layer canvas | GT crop | |||||||
|---|---|---|---|---|---|---|---|---|
| Autoencoder / variant | Layered | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | rFID |
| LayerDiffuse RGBA | 42.04 | 0.9749 | 0.01706 | 28.97 | 0.8701 | 0.04129 | 2.432 | |
| AlphaVAE (FLUX) | 35.57 | 0.9736 | 0.02773 | 27.70 | 0.8833 | 0.05287 | 30.669 | |
| QIL RGBA-VAE | 49.59 | 0.9911 | 0.00324 | 37.83 | 0.9744 | 0.00659 | 1.477 | |
| QIL + adapted decoder | 54.94 | 0.9956 | 0.00391 | 41.92 | 0.9882 | 0.00683 | 0.326 | |
| ProtoVAE AdaLN-Zero decoder | 51.90 | 0.9943 | 0.00499 | 39.17 | 0.9842 | 0.00862 | 1.042 | |
| RGB L1 | Alpha soft IoU | |||||||||||
| # max allowed layer merges | 0 | 1 | 2 | 3 | 4 | 5 | 0 | 1 | 2 | 3 | 4 | 5 |
| VLM Base Hi-SAM | 0.120 | 0.103 | 0.089 | 0.081 | 0.076 | 0.073 | 0.560 | 0.630 | 0.686 | 0.722 | 0.747 | 0.759 |
| YOLO Base Hi-SAM | 0.096 | 0.083 | 0.071 | 0.063 | 0.059 | 0.058 | 0.570 | 0.654 | 0.717 | 0.757 | 0.781 | 0.790 |
| LayerD | 0.071 | 0.054 | 0.046 | 0.042 | 0.040 | 0.040 | 0.752 | 0.811 | 0.844 | 0.856 | 0.862 | 0.865 |
| QIL | 0.059 | 0.049 | 0.039 | 0.038 | 0.036 | 0.036 | 0.871 | 0.886 | 0.911 | 0.912 | 0.916 | 0.916 |
| CLD | 0.081 | 0.075 | 0.063 | 0.060 | 0.055 | 0.053 | 0.822 | 0.857 | 0.897 | 0.913 | 0.929 | 0.935 |
| Merged RGB | Prompt and runtime | |||||
|---|---|---|---|---|---|---|
| Method | Layered | SSIM | FID | Align. | OCR | Time (s) |
| QI | 0.5904 | 98.08 | 0.984 | 0.850 | 19.31 | |
| QI + QIL | — | — | 0.984 | 0.840 | 230.80 | |
| FLUX | 0.5925 | 86.84 | 0.848 | 0.460 | 9.63 | |
| FLUX + QIL | — | — | 0.832 | 0.470 | 220.86 | |
| ProtoDiffusion | 0.6898 | 66.41 | 0.940 | 0.860 | 11.58 | |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Sample | VLM axis / outcome | Observed decomposition behavior |
|---|---|---|
| sample_0005 | Text + non-text: 5/5 | Text (“The”, “KOL”, “Engagement”, “workshop”) is isolated in individual layers with clean alpha; graphic foreground is removed from the white background without visible remnants. |
| sample_0007 | Non-text + inpainting: 5/5 | The forest scene remains in the background while the alligator, moon, trees, and roots are assigned to separate layers. The occluded background regions are described as semantically consistent, without holes or smears. |
| sample_0357 | Text: 0/5 | The large title (boxes 3–4) is absent from the foreground layers and survives only as a faint ghost in the background, with no dedicated text layers. |
| sample_0063 | Non-text: 0/5 | One decoded layer is a near-complete copy of the input, the background is a solid color rather than the scene, and the other decoded layers contain only small, misplaced fragments. |
| sample_0092 | Non-text + inpainting: 0/5 | Most decoded layers are empty; the remaining ones contain fragmented blue lines that do not match the supplied boxes. Since foreground extraction fails, the background is empty rather than inpainted. |
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 507 papers