Generalization in generative modeling is defined as the ability to learn an underlying distribution from a finite dataset and produce novel samples, with evaluation largely driven by held-out performance and perceived sample quality. In practice, training is often stopped at the minimum of the test loss, taken as an operational indicator of generalization. We challenge this viewpoint by identifying a phase of biased generalization during training, in which the model continues to decrease the test loss while favoring samples with anomalously high proximity to training data. By training the same network on two disjoint datasets and comparing the mutual distances of generated samples and their similarity to training data, we introduce a quantitative measure of bias and demonstrate its presence on real images. We then study the mechanism of bias, using a controlled hierarchical data model where access to exact scores and ground-truth statistics allows us to precisely characterize its onset. We attribute this phenomenon to the sequential nature of feature learning in deep networks, where coarse structure is learned early in a data-independent manner, while finer features are resolved later in a way that increasingly depends on individual training samples. Our results show that early stopping at the test loss minimum, while optimal under standard generalization criteria, may be insufficient for privacy-critical applications.
Figures & tables
Figure 1: Biased generalization emerges before overfitting across models and settings. (a) Sample-split analysis on CelebA: we compare two denoising diffusion models trained on non-overlapping data slices. Left: cosine distance between generated samples (left axis) and denoising score matching (DSM) test loss (right axis) during training, means over 15 models with standard errors. Generated samples become maximally similar before the test loss is minimal, indicating the onset of biased generalization while the test loss is still decreasing. Colored stars mark epochs selected for visualization. Right: samples generated at the starred epochs by a model trained on each of the database split A/B (central columns), with nearest neighbors (NN) from each training split (side columns). Early in training, both models evolve similarly and sample quality improves; near the test-loss minimum, generated samples can differ substantially across splits and may get close to training examples, showing a bias without exact memorization. (b) Neural network trained on a controlled hierarchical dataset. Model-oracle divergence (left axis), representing the distance of the learned score to the exact score (red) and four coarser versions of the latter that account for lower-level features (as indicated by arrows). Sample-split analysis (right axis, blue) measuring the distance between denoising scores of models trained on disjoint datasets. All scores are computed on test samples (w.r.t the model’s training data), noised up to a critical diffusion time t/T=0.15 . The biased generalization phase is seen between the minimum of the sample-split curve (blue) and the minimum of the model-exact oracle divergence (red). It takes place when the models start resolving finer-scale features (light green). (c) Training-free score model on the same hierarchical data, parametrized by a sharpness parameter ε controlling the concentration of probability mass around the training data. Top: divergence between generated data and ground truth distributions of distances to the nearest neighbor (NN) in the training set (left axis) and DSM test loss (right axis) as a function of the sharpness parameter ε , showing sizeable bias at the test-loss minimum. Stars indicate selected values of ε . Bottom: in-training NN overlap (1 - normalized distance) distributions at selected sharpness levels, comparing model samples (solid black) to the ground truth (dashed gray).
Figure 2: Cosine distance between the predictions of two networks trained on disjoint subsets of CelebA of size n=1024 , evaluated on inputs noised until time t out of T=1000 . The distance is evaluated for original images that are either outside of both training sets (“Test”) or in one of them (“Train”). The vertical dashed lines correspond to the minima of the diffusion time-averaged metrics shown in Fig. 1 (a).
Figure 3: (a) Illustration of an ℓ=3 tree-based data generation for the full model (top) and a hierarchically filtered version with k=2 (bottom). (b) Kullback-Leibler divergence between the exact posterior mean of BP 0 and those from filtered BP k denoisers along a reverse trajectory following the exact BP, averaged over 2k realizations. Dashed lines show trained models at the minimum of their test losses for different training set size n .
Figure 4: Bias metrics for a diffusion model trained on n=12k tree-based sequences computed with 50 k generated samples and evaluation points, and averaged over 15 training runs and showing the standard error. (a) Nearest-neighbor divergence (Sec. 2.2 ) of generated sequences and denoising test loss as a function of training. (b) Expectation value of the excess data-dependent loss of (4) evaluated on test and train sequences noised up to time t=150 . Vertical dashed lines show the minimum of the bias metric (blue) and test loss (red).
Figure 5: Average normalized overlap to the starting sequence for a “U-turn” experiment after noising to time t using 100 trajectories for each 1000 starting points, model trained on n=12k data. Left: checkpoint minimizing the Kullback-Leibler divergence of nearest neighbor overlaps with the result expected from fair sampling. Right: checkpoint minimizing the denoising test loss. Shaded areas show the standard error over starting sequences.
Figure 6: Evolution of model divergences DKL(⋅∣∣x^0θ) for different BP k (from purple to green, decreasing in k ) and BP 0 (red), evaluated on models A ( −∙ ) and B ( −− ). Blue curves ( −▲ ) show the sample-split denoising score divergence DKL(x^0θA∣∣x^0θB) . Left: n=5k , Right: n=70k .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Set of the 20 images among 512 with the closest to typical cosine similarities between scores after (from left to right) 111, 213, 620 and 1333 epochs. As in Fig. 1 (a) (fourth row from the bottom here), central columns show samples generated by two models given the same noise trajectories, while outer columns show the closest image in their respective training sets. On the left side of each row is displayed the cosine similarity between the generated images, while below outer images is displayed the cosine similarity between the generated images and the associated closest training sample.
Figure 8: Reproduction of the experiment presented in Fig. 2 for two additional fixed diffusion times.
Figure 9: Left: NN divergence and DSM test loss versus training epoch for a model trained on n=12k samples of hierarchical data, with respective minima indicated by vertical dotted lines. Right: U‑turn overlap ratios across epochs for starting points in the training set of the model (black lines) or from a test set (red dashed lines). Blue/red framed panels highlight the epochs corresponding to the bias metric minimum and test‑loss minimum respectively (shown in Fig. 5 ).
Figure 10: Left: NN divergence and DSM test loss versus training epoch for a model trained on n=5k samples of hierarchical data, with respective minima indicated by vertical dotted lines. Right: u‑turn overlap ratios across epochs for starting points in the training set of the model (black lines) or from a test set (red dashed lines). Blue/red framed panels highlight the epochs corresponding to the bias metric minimum and test‑loss minimum respectively.
Figure 11: U-turn experiment performed by noising sequences whose discrete elements are drawn uniformly at random. The left y-axis shows the averaged overlap of the generated sequences with the original ones.
This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the model actually learns during pre-memorization phase. To highlight our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.
Pierre Marion, Yu-Han Wu
Inria, École Normale Supérieure, PSL Research University · LPSM, Sorbonne University & Google DeepMind
Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.
Tyler Farghly, Benjamin Dupuis, Alain Durmus +1
1INRIA - École Normale Supérieure - PSL Research University - CNRS, France · 2 École Polytechnique - CMAP - IP Paris, France
Conditional diffusion models generate diverse, novel, and high-quality samples under prescribed conditions. However, theoretical understanding of their memorization and generalization remains limited, while recent works have characterized these behaviors primarily in unconditional settings. In this work, we analyze a random-feature conditional score model in the high-dimensional proportional limit, deriving asymptotic expressions for training and test losses. By decomposing the test loss, we show that in the overparameterized regime, increasing model width improves prediction of the condition-dependent mean while reducing within-condition prediction variance, a phenomenon we term "malign generalization." Furthermore, analyzing the training loss reveals that more informative conditions lead to memorization of training samples at smaller widths. These theoretical findings are supported by experiments with U-Net architectures on realistic data.
Gwangho Kim, Sungyoon Lee
Department of Computer Science Hanyang University, Seoul