cs.CVNov 17, 2026

Do Diffusion Models Learn to Generalize Basic Visual Skills?

Authors: Amish Sethi, Boya Zeng, Wenhao Chai, Zhuang Liu

Organizations: Princeton University

Abstract

Diffusion models generate high-quality images, but how well they generalize is hard to measure. Current evaluations rely on human preference and prompt-following benchmarks for text-to-image models and on distribution-level metrics such as Fréchet Inception Distance (FID) and Inception Score (IS) for class-conditional models, but these cannot tell whether models learn basic visual skills or largely match training patterns. To address this gap, we introduce the Visual Generative Lab (VGL), a controlled experimental framework for studying how diffusion models learn and generalize basic visual skills, including size, position, rotation, count, color, shape, and their compositions. By generating synthetic training data for each skill from explicit rules, VGL allows precise measurement of generalization with task-specific, rule-based metrics. Training models from scratch on these data reveals three patterns in this controlled setting: (1) with our default design, extrapolation is limited: size and position degrade within a few pixels of the training range, count reaches only about 50% accuracy on out-of-range counts, and rotation extrapolation accuracy varies widely across seeds; (2) for compositional generalization, the number of distinct basic visual skill combinations matters more than dataset size; (3) skills mostly fail independently, except position. Code and data-generation scripts: https://github.com/AmishSethi/visual-generative-lab.

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives

    Oct 3, 2025Karim Farid, Rajat Sahay, Yumna Ali Alnaggar +4Compositional Image GenerationGenerative Modeling

  2. When Do Diffusion Models learn to Generate Multiple Objects?

    Apr 30, 2026Yujin Jeong, Arnas Uselis, Iro Laina +2T2I GenerationCompositional Image Generation

  3. A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic

    Apr 20, 2026Joon Hyeok Kim, Yong-Hyun Park, Mattis Dalsætra Østby +1Pixel-Space Diffusion ModelsMechanistic Interpretability