ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation
Organizations: New York University
Abstract
In this paper, we propose ProCreate, a simple and easy-to-implement method to improve sample diversity and creativity of diffusion-based image generative models and to prevent training data reproduction. ProCreate operates on a set of reference images and actively propels the generated image embedding away from the reference embeddings during the generation process. We propose FSCG-8 (Few-Shot Creative Generation 8), a few-shot creative generation dataset on eight different categories -- encompassing different concepts, styles, and settings -- in which ProCreate achieves the highest sample diversity and fidelity. Furthermore, we show that ProCreate is effective at preventing replicating training data in a large-scale evaluation using training text prompts. Code and FSCG-8 are available at https://github.com/Agentic-Learning-AI-Lab/procreate-diffusion-public. The project page is available at https://procreate-diffusion.github.io.
Figures & tables
| Subset | Method | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|---|
| Amedeo | DDIM | 16.71 0.39 | 2.39 0.10 | 0.75 0.09 | 0.36 0.06 | 0.34 0.01 | 14.89 0.69 | 0.35 0.00 |
| CADS | 12.83 0.48 | 2.29 0.12 | 0.73 0.05 | 0.39 0.02 | 0.34 0.01 | 15.17 0.40 | 0.35 0.00 | |
| ProCreate | 9.28 0.39 | 1.59 0.25 | 0.57 0.05 | 0.65 0.05 | 0.26 0.01 | 22.20 0.55 | 0.35 0.00 | |
| Apple | DDIM | 48.51 3.84 | 1.37 0.13 | 0.51 0.03 | 0.75 0.07 | 0.17 0.01 | 25.13 0.69 | 0.30 0.00 |
| CADS | 38.88 1.47 | 1.17 0.22 | 0.43 0.04 | 0.79 0.01 | 0.17 0.01 | 26.34 0.87 | 0.30 0.00 | |
| ProCreate | 24.59 0.14 | 0.62 0.10 | 0.45 0.06 | 0.81 0.02 | 0.12 0.00 | 32.33 0.37 | 0.29 0.00 |
| Subset | Method | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|---|
| Amedeo | DDIM | 17.38 0.40 | 2.96 0.13 | 0.48 0.04 | 0.63 0.15 | 0.28 0.00 | 14.12 0.37 | 0.32 0.00 |
| CADS | 16.19 0.56 | 2.56 0.17 | 0.53 0.06 | 0.65 0.07 | 0.27 0.00 | 14.37 0.40 | 0.32 0.00 | |
| ProCreate | 8.88 0.17 | 1.19 0.27 | 0.45 0.05 | 0.80 0.01 | 0.20 0.01 | 25.49 0.55 | 0.31 0.00 | |
| Apple | DDIM | 29.71 0.63 | 0.70 0.07 | 0.39 0.02 | 0.71 0.05 | 0.22 0.01 | 17.38 1.04 | 0.27 0.00 |
| CADS | 32.52 2.21 | 0.84 0.12 | 0.35 0.06 | 0.71 0.02 | 0.21 0.01 | 18.82 1.39 | 0.27 0.00 | |
| ProCreate | 20.12 2.44 | 0.27 0.05 | 0.40 0.03 | 0.84 0.04 | 0.15 0.00 | 28.23 0.83 | 0.27 0.00 |
| MSLA | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|
| 0 | 35.11 2.60 | 4.06 0.48 | 0.69 0.06 | 0.71 0.05 | 0.18 0.00 | 26.15 0.56 | 0.34 0.00 |
| 1 | 29.99 1.49 | 2.64 0.08 | 0.60 0.06 | 0.91 0.05 | 0.13 0.01 | 30.90 0.69 | 0.33 0.00 |
| 3 | 17.82 2.99 | 0.79 0.47 | 0.64 0.02 | 0.91 0.04 | 0.10 0.01 | 33.51 0.58 | 0.33 0.00 |
| 5 | 14.10 1.64 | 0.72 0.11 | 0.66 0.06 | 0.97 0.02 | 0.09 0.00 | 34.12 0.36 | 0.33 0.00 |
| Sampler | Method | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|---|
| DDIM | Default | 35.11 2.60 | 4.06 0.48 | 0.69 0.06 | 0.71 0.05 | 0.18 0.00 | 26.15 0.56 | 0.34 0.00 |
| CADS | 34.39 1.97 | 4.27 0.41 | 0.70 0.07 | 0.81 0.06 | 0.18 0.00 | 25.97 0.20 | 0.34 0.00 | |
| ProCreate | 14.10 1.64 | 0.72 0.11 | 0.66 0.06 | 0.97 0.02 | 0.10 0.00 | 33.51 0.36 | 0.33 0.00 | |
| DDPM | Default | 37.29 1.97 | 4.08 0.27 | 0.68 0.05 | 0.70 0.07 | 0.19 0.01 | 26.05 0.68 | 0.34 0.00 |
| CADS | 37.24 1.81 | 4.36 0.31 | 0.68 0.03 | 0.73 0.08 | 0.19 0.01 | 25.62 0.48 | 0.34 0.00 | |
| ProCreate | 14.80 2.18 | 2.03 0.22 | 0.67 0.08 | 0.95 0.03 | 0.12 0.01 | 31.74 0.62 | 0.34 0.00 |
| Subset | Method | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|---|
| Amedeo | DDIM | 11.27 0.69 | 2.02 0.18 | 0.81 0.03 | 0.44 0.12 | 0.36 0.01 | 14.32 0.70 | 0.34 0.00 |
| CADS | 12.51 0.68 | 2.43 0.22 | 0.80 0.04 | 0.39 0.08 | 0.36 0.02 | 14.78 0.96 | 0.34 0.00 | |
| ProCreate | 10.23 0.66 | 0.21 0.03 | 0.69 0.06 | 0.55 0.17 | 0.19 0.01 | 30.46 0.84 | 0.33 0.01 | |
| Apple | DDIM | 19.68 2.39 | 0.36 0.13 | 0.63 0.04 | 0.70 0.12 | 0.11 0.00 | 36.11 0.34 | 0.30 0.00 |
| CADS | 27.83 1.34 | 0.83 0.11 | 0.48 0.02 | 0.71 0.12 | 0.11 0.01 | 36.65 0.75 | 0.29 0.00 | |
| ProCreate | 14.55 1.49 | 0.12 0.03 | 0.64 0.05 | 0.78 0.10 | 0.11 0.01 | 36.70 0.63 | 0.30 0.00 |
| Subset | Method | FID | KID | Precision | Recall | MSS | Vendi | Prompt Fid. |
|---|---|---|---|---|---|---|---|---|
| Amedeo | DDIM | 14.87 1.42 | 3.01 0.37 | 0.66 0.07 | 0.69 0.19 | 0.30 0.01 | 14.07 0.41 | 0.34 0.00 |
| CADS | 16.51 0.98 | 3.48 0.32 | 0.58 0.08 | 0.57 0.19 | 0.32 0.01 | 13.68 0.26 | 0.34 0.00 | |
| ProCreate | 12.22 1.04 | 2.89 0.24 | 0.41 0.07 | 0.75 0.05 | 0.29 0.01 | 16.03 0.30 | 0.33 0.01 | |
| Apple | DDIM | 45.35 2.07 | 2.77 0.22 | 0.58 0.03 | 0.76 0.08 | 0.18 0.01 | 17.85 0.54 | 0.31 0.00 |
| CADS | 30.89 0.35 | 2.73 0.09 | 0.58 0.08 | 0.78 0.05 | 0.19 0.01 | 17.13 0.52 | 0.31 0.00 | |
| ProCreate | 25.16 1.07 | 2.48 0.14 | 0.58 0.05 | 0.80 0.08 | 0.14 0.01 | 20.01 0.64 | 0.31 0.00 |
Image Generation
Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.
See all 501 papers