Do Diffusion Models Learn to Generalize Basic Visual Skills?
Organizations: Princeton University
Abstract
Diffusion models generate high-quality images, but how well they generalize is hard to measure. Current evaluations rely on human preference and prompt-following benchmarks for text-to-image models and on distribution-level metrics such as Fréchet Inception Distance (FID) and Inception Score (IS) for class-conditional models, but these cannot tell whether models learn basic visual skills or largely match training patterns. To address this gap, we introduce the Visual Generative Lab (VGL), a controlled experimental framework for studying how diffusion models learn and generalize basic visual skills, including size, position, rotation, count, color, shape, and their compositions. By generating synthetic training data for each skill from explicit rules, VGL allows precise measurement of generalization with task-specific, rule-based metrics. Training models from scratch on these data reveals three patterns in this controlled setting: (1) with our default design, extrapolation is limited: size and position degrade within a few pixels of the training range, count reaches only about 50% accuracy on out-of-range counts, and rotation extrapolation accuracy varies widely across seeds; (2) for compositional generalization, the number of distinct basic visual skill combinations matters more than dataset size; (3) skills mostly fail independently, except position. Code and data-generation scripts: https://github.com/AmishSethi/visual-generative-lab.
Figures & tables
| Skill | Type | Domain | Extrapolation values |
|---|---|---|---|
| size | continuous | radius px | px |
| position | continuous (2D) | px | px outside (320 positions) |
| rotation | continuous | angle | |
| count | discrete | objects | |
| shape | categorical | 4 classes | — |
| color | categorical | 8 classes | — |
| size | position | rotation | count | |||||||||
| Model | Train | Interp. | Extrap. | Train | Interp. | Extrap. | Train | Interp. | Extrap. | Train | Interp. | Extrap. |
| DiT-S/2 (baseline) | — | |||||||||||
| + sinusoidal emb. | — | |||||||||||
| + rotary emb. | — | |||||||||||
| + linear emb. | — | |||||||||||
| + AdaLN | — | |||||||||||
| Visual Skills | Cov. | Cat. | A (%) | shape (%) | color (%) | position (%) | size (%) | count (%) | rotation (%) |
| color + shape | 25% | B | 100.0 | — | — | — | — | ||
| 50% | B | 100.0 | — | — | — | — | |||
| 75% | B | 100.0 | — | — | — | — | |||
| color + count | 25% | B | 100.0 | — | {\color[rgb]{0.4,0.4,0.4}79.9_{\pm 22.1}} | — | — | {\color[rgb]{0.4,0.4,0.4}38.2_{\pm 42.2}} | — |
| 50% | B | 100.0 | — | — | — | {\color[rgb]{0.4,0.4,0.4}37.7_{\pm 3.5}} | — | ||
| 75% | B | 100.0 | — | — | — | — |
| Split | size | position | shape | color |
|---|---|---|---|---|
| In range ( B ) | 100.0 | 100.0 | 100.0 | 100.0 |
| size OOD ( C ()) | 54.6 | 100.0 | 100.0 | 100.0 |
| position OOD ( C ()) | 79.5 | 0.0 | 32.0 | 100.0 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Visual Skill | Type | Range / Values | Value grid |
|---|---|---|---|
| size | Continuous | pixels | 16 values (step 1.0) |
| position | Continuous (2D) | grid (1024 positions) | |
| rotation | Continuous | 271 values (step 1 ∘ ) | |
| count | Discrete | objects | 6 values |
| shape | Categorical | circle, square, triangle, diamond | 4 classes |
| color | Categorical | red, blue, green, yellow, magenta, cyan, orange, purple | 8 classes |
| MAE ( ∘ ) | Acc. (%) | |||||
| Training angles | Step | Train | Interp. | Extrap. | Extrap. | |
| to | 300 | 0.30 | 0.53 | 0.87 | 100.0 | |
| to | 10 | 0.10 | 10.6 | 26.4 | 11.2 | |
| to | 9 | 0.00 | 2.6 | 47.9 | 14.7 | |
| to | 180 | 0.17 | 0.51 | 44.5 | 45.9 | |
| to | 12 | 0.08 | 2.4 | 98.4 | 22.8 | |
| Train | Interp. | Extrap. | ||||
| Seed | MAE ( ∘ ) | Acc. @ (%) | MAE ( ∘ ) | Acc. @ (%) | MAE ( ∘ ) | Acc. @ (%) |
| 0 | 0.31 | 100.0 | 0.54 | 100.0 | 8.41 | 63.2 |
| 1 | 0.30 | 100.0 | 0.53 | 100.0 | 48.94 | 53.2 |
| 2 | 0.29 | 100.0 | 0.53 | 100.0 | 1.90 | 99.9 |
| 3 | 0.27 | 100.0 | 0.53 | 100.0 | 1.49 | 100.0 |
| 4 | 0.32 | 100.0 | 0.54 | 100.0 | 0.55 | 100.0 |
| size | position | rotation | count | |
| range width | 15 px | 36 px | 5 | |
| — | — | |||
| — | ||||
| — | — | |||
| — | — | |||
| — | — |
| skill | 10k | 20k | 40k | 80k |
|---|---|---|---|---|
| size | ||||
| position | ||||
| rotation | ||||
| count |
| space | model | tokens | size | rotation |
|---|---|---|---|---|
| latent | DiT-S/2 | 16 | 7.1 | 21.2 |
| pixel | DiT-S/16 | 16 | — | 2.7 |
| latent | DiT-S/1 | 64 | 36.1 | 18.5 |
| pixel | DiT-S/2 | 1024 | 51.5 | 68.0 |
| pixel | VAE latent | |||||
|---|---|---|---|---|---|---|
| skill | train | interp | extra | train | interp | extra |
| size | ||||||
| position | ||||||
| rotation | ||||||
| count | — | — | ||||
| skill | train | interp | extra |
|---|---|---|---|
| size | |||
| rotation | |||
| count | — |
| skill | simple | textured | |
|---|---|---|---|
| size | 77.3 | 56.9 | |
| position | 66.5 | 67.7 | +1.2 |
| rotation | 66.0 | 86.3 | +20.3 |
| count | 12.9 | 15.4 | +2.5 |
| model | count | position | size | rotation |
|---|---|---|---|---|
| Stable Diffusion 1.5 [ 53 ] | 8.6 | 11.5 | 24.6 | 16.4 |
| SDXL [ 50 ] | 1.0 | 15.7 | 11.4 | 13.1 |
| PixArt- [ 11 ] | 32.0 | 11.1 | 31.7 | 13.8 |
| PixArt- [ 10 ] | 32.5 | 11.1 | 26.7 | 13.8 |
| HunyuanDiT [ 36 ] | 30.0 | 11.1 | 22.8 | 18.1 |
| Shuttle 3 [ 58 ] | 49.2 | 61.2 | 16.7 | 31.7 |
| coverage | seeds | held-out (%) | ||||
|---|---|---|---|---|---|---|
| 4 | 8 | 25% | 8 | 2 | 3 | |
| 8 | 8 | 25% | 16 | 2 | 3 | |
| 4 | 16 | 25% | 16 | 4 | 3 | |
| 8 | 16 | 25% | 32 | 4 | 1 | 100.0 |
| 4 | 8 | 50% | 16 | 4 | 3 | |
| 8 | 8 | 50% | 32 | 4 | 3 |
| Model | Split | size (%) | position (%) | shape (%) | color (%) |
|---|---|---|---|---|---|
| 3-skill, run 1 | A (training) | 100.0 | 100.0 | 100.0 | — |
| B (in-range) | 99.6 | 100.0 | 100.0 | — | |
| C () ( size OOD) | 37.6 | 100.0 | 100.0 | — | |
| C () ( position OOD) | 30.9 | 0.0 | 20.4 | — | |
| D (both OOD) | 6.6 | 0.0 | 25.5 | — | |
| 3-skill, run 2 | A (training) | 100.0 | 100.0 | 100.0 | — |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,266 papers