Diffusion models trained on distinct datasets can produce highly similar samples when prompted with the same noise input, suggesting that they learn a generalized data distribution. We investigate the role of noise injected during the sampling process, and show that it plays an important role in making or breaking this strong generalization effect. Based on this finding we investigate the robustness of sample similarity to changes in the injected noise, and find that its impact is largest in the sampling regime where broad structure already exists but the precise identity of the sample is not yet determined.
Figures & tables
Figure 1: Top : Exemplary denoised sampling trajectories (rows) using different injected noise. Bottom : The sensitivity to changes in the injected noise sequence is significantly higher during regime II than regime I and III ( p -values show two-sided paired t -tests for adjacent intervals, α=0.05 ).
Figure 2: Left : Evaluating the impact of pretrained model, starting noise and injected noise sequence. Generated images are reliably robust against changes in starting noise and which pretrained model is used for sampling. The injected noise sequence has the largest impact in determining the final image. Exemplary images for one trial are shown. Right : To quantitatively asses consistency we report the average cosine similarity for configurations sharing/not-sharing the injected noise sequence. Configurations sharing the injected noise sequence produce highly consistent images, whereas using different noise sequences leads to substantially different results.
Figure 3: Left : Cosine similarity between intermediate states quickly rose in the beginning of the sampling process. After 10% denoising steps it reached ≈1.0 , staying high until sampling finished. The model could already determine the direction early on, indicated by high similarity in the beginning for x^t . Right : When the injected noise sequence was not shared the consistency stayed at ≈0.0 during sampling. However for x^t similarity was initially high and then quickly dropped as sampling continued. We attribute this to the model producing general coarse face features early on.
Figure 4: Generalization in diffusion models is not only dependent on starting noise. Beginning from two random starting noise conditions, intermediate states of the sampling process are shown. Similarity between samples is initially low but quickly rises. In the end both starting noise samples generate an almost identical face.
Figure 5: We show sampling trajectories for two configurations. The top two rows show intermediate states xt , with the first row using model A and the second row using model B, each with their own respective random starting noise. The bottom two rows show the trajectory for the same configuration and sampling process but with x^t , i.e. states where no injected noise is added to visually highlight the consistency between trajectory states.
Figure 6: Generalization between model A and model B is also possible when using a much smaller dataset (500 images each) as long as the same injected noise sequence is used during DDPM sampling. This result aligns with the findings from above (see Sec. 3.1 ).
Figure 7: Left : Similarity rises during sampling; however, in this setup, it takes longer for x^t , as the model is likely less capable. Right : When different noise is injected, no similarity arises. Even in early states, the model is not able to pull in similar general directions for x^t .
Figure 8: Sampling trajectories for model A and model B are shown. The top two rows show xt , and the bottom two rows show x^t , for the same injected noise with different starting noise conditions between model A and model B.
Figure 9: Top : Exemplary DM sampling trajectories (rows) using different injected noise sequences for the butterfly models. Bottom : The sensitivity to changes in the injected noise sequence rises throughout the process, early states do not show increased robustness, likely limited by the models capabilities ( p -values show two-sided paired t -tests for adjacent intervals, α=0.05 ).
Figure 10: We initialize denoising with an increasing ratio between a clean image from the dataset and noise. Depicted are the cosine similarities to the clean image x∗ and the image that would be generated from zstart , with fixed injected noise. The dashed lines show exemplary similarities of individual samples (where the clean image is the same across different α ).
Figure 11: The average diffusion coefficient σt during sampling, computed dynamically according to Kadkhodaie and Simoncelli [2020] . This coefficient regulates the amount of injected noise based on the Signal-to-Noise-Ratio of the current state.