stat.MLMar 13, 2026

A theory of learning data statistics in diffusion models, from easy to hard

Authors: Lorenzo Bardone, Claudia Merger, Sebastian Goldt

Organizations: Statistical Physics of Computation Laboratory, EPFL, Lausanne, Switzerland · Theoretical and Scientific Data Science Group, SISSA, Trieste, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.

Figures & tables

Figure 1: Sequential learning in diffusion models. a) Test loss of diffusion model and loss on CIFAR-10 clones during training. Vertical dotted lines mark training stages of images generated from the model shown in panels d)-g). All curves are averages over 3 initializations of the network models and 5⋅1035\cdot 10^{3} test data. Shaded areas report standard deviation over random initialization. Panel b) reports the same as a), but for denoising samples with fixed level of noise x=e−tx0+1−e−2tzx=e^{-t}x_{0}+\sqrt{1-e^{-2t}}z where x0x_{0} is a data point and noise z∼N(0,Id)z\sim{\mathcal{N}\left(0,\text{Id}\right)} . c) Test loss of a neural network trained on the mixed cumulant model with dimension d=102d=10^{2} a fixed level of noise, evaluated on clones of the data set. All curves in panel c) are averages over 55 initializations of the network model and 10410^{4} test data. d)-g) Samples generated from U-nets [ Ronneberger et al., 2015 ] on CIFAR-10 at various training stages.
Figure 2: Examples of the contraction term Λ\Lambda for different choices of activation σ\sigma . σ∗\sigma^{*} denotes the matched functional form of the score eq. 49 for different values of the diffusion time tt .
Figure 3: Normalized overlap of first-layer weights of neural networks of varying depth trained with Adam on inputs drawn from the mixed cumulant model, eq. 6 , at d=100d=100 . All curves are averages over 55 random initializations of the neural networks.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Samples from the different "clones" as well as the test data set. a) shows images drawn from the mean clone which follows a Gaussian distribution with matching mean to the CIFAR-10 dataset and identity covariance. In b), we additionally match the covariance matrix of the Gaussian distribution to the CIFAR-10 dataset. c) shows 9 images from the CIFAR-10 dataset.
Figure 5: Samples from the different "clones" as well as the test data set. a) shows images drawn from the mean clone which follows a Gaussian distribution with matching mean to the CelebA dataset and identity covariance. In b), we additionally match the covariance matrix of the Gaussian distribution to the CelebA dataset. c) shows 9 images from the CelebA dataset.
Figure 6: Training Unets on CelebA data. a)-d) samples generated from the model at various training stages. e) Test loss of the model on clones of the dataset during training. Vertical dotted lines mark training stages of images generated from the model shown in panels a)-d). f)-g) same as e), but for fixed level of noise. All curves are averaged over 2 initializations of the network models and 5⋅1035\cdot 10^{3} test data. Shaded areas report standard deviation over random initialization.
Figure 7: Normalized overlap of first-layer weights of neural networks of varying mm trained with Adam or SGD the MCM model at d=100d=100 . All curves are averages over 55 random initializations of the neural networks.
Figure 8: Normalized overlap of first-layer weights with cumulant spike of neural networks trained with SGD on inputs drawn from the mixed cumulant model over training steps, divided by the (cubed) dimension dd . a) shows the case of a single cumulant spike, with number of steps scaled by d−3d^{-3} . Panel b) shows the case of an MCM model with a covariance and a cumulant spike, which have correlated latents, with training steps scaled by d−1d^{-1} . All curves are averages over 55 random initializations of the neural networks, with two-layer autoencoder architecture and ReLu nonlinearity in the overparametrized regime.
More in this topic

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,169 papers

Explore similar work

CardsList

  1. Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

    Mar 24, 2026Anand Jerry George, Nicolas MacrisIntrinsic DimensionalityRandom Feature Methods

    École Polytechnique Fédérale de Lausanne (EPFL), Lab for Statistical Mechanics of Inference in Large Systems (SMILS), CH-1015 Lausanne, Switzerland

  2. Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions

    Sep 4, 2024Peng Wang, Huijie Zhang, Zekai Zhang +3Intrinsic DimensionalitySample Complexity

    University of Macau · University of Michigan · University of California, Berkeley

  3. Diffusion Models Are Statistically Optimal for Learning Low-Dimensional Multi-Modal Distributions

    May 28, 2026Jingda Wu, Changxiao CaiSample ComplexityStatistical Learning Theory

    Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA.