Emergence of Distortions in High-Dimensional Guided Diffusion Models
Authors: Enrico Ventura, Beatrice Achilli, Luca Ambrogioni, Carlo Lucibello
Organizations: Department of Computing Sciences, Bocconi University, Milan, Italy · Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands · Bocconi Institute for Data Science and Analytics, Bocconi University, Milan, Italy
Classifier-free guidance (CFG) is the de facto standard for conditional sampling in diffusion models, yet it often reduces sample diversity. Using tools from statistical physics, we analyze the emergence of generative distortions induced by CFG, namely the mismatch between the CFG sampling distribution and the true conditional distribution. We study this phenomenon in analytically tractable settings with exact score functions, characterizing its dependence on data dimensionality and the number of classes. For high-dimensional Gaussian mixtures, we use dynamic mean-field theory to show that distortions persist when the number of classes scales exponentially with the data dimension, whereas they vanish in the sub-exponential regime due to a dynamical phase transition. Moreover we found that, in the infinite-class limit, distortions remain unavoidable regardless of dimensionality because of the increasing density of classes. Finally, we show that standard CFG schedules cannot prevent variance shrinkage, and we propose a theoretically grounded guidance schedule incorporating a negative-guidance window that improves both class separability and sample diversity in real-world latent diffusion models.
Figures & tables
Figure 1: Speciation time ts and the distortion estimators in the exponential regime predicted by the theory as functions of the control parameters β=log(M)/d and w , for σ2=0.5 . In the white region, there is no speciation and therefore we have no transition to the conditional phase. This regime displays strong distortion in the conditional sampling as testified by the behavior of δμ and δσ2 . In the small β regime instead, where ts is larger, distortion is weak.
Figure 2: Comparison of the mean-field prediction (full lines) with numerical simulations of CFG (circles) for a mixture of an exponential number of Gaussians as data distribution. The plots show the behavior of distortion estimators δμ and δσ2 , errors are standard deviations of the mean. Theory matches qualitatively and also quantitatively the numerics at increasing d .
Figure 3: (a) Measures of distortion from a guided Stable Diffusion (v1.5) model, in feature space, as a function of the guidance level w , averaged over 50 prompts × 20 samples, error bars are standard deviations of the mean. Blue circles refer to CLIP feature extractor, yellow ones to DINOv2. (b) Samples generated from the prompt a fantasy landscape with castles and dragons, vibrant colors, digital art . Rows refer to different random seeds, columns to guidance levels.
Figure 4: (a): Distortions’phase diagram for the negative-window schedule, as function of w0 and ω . Data are from the Gaussian Mixture model with σ2=0.75 and β>2 . The orange region signals a gain in both sample diversity and class separability, other regions show either loss of separability or tendency to shrink the mode variance. (b): We compare several guidance schedules: our negative-window prescription is consistent with standard techniques in prompt alignment and outperforms them in sample diversity. Points are averaged over 50 prompts × 20 samples, error bars are standard deviations of the mean.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Ref.
Gaussian
GMM
GMM
General &
Finite d
d→∞
Class
Diversity
(linear score)
M=O(1)
M scaling with d
Statistical Theory
Alignment
[ 31 ] , Li et al. (2025)
x
x
x
[ 37 ] , Pavasovic et al. (2025)
x
x
x
x
[ 24 ] , Jin et al. (2025)
x
x
x
x
[ 30 ] , Li & Jiao (2025)
x
[ 10 ] , Chidambaram et al. (2024)
x
x
x
x
Appendix
Table 1: Comparison of data models, dimensional assumptions, and core analytical objectives in theoretical studies of guided diffusion models.
Figure 5: CFG induced distortions from theory and numerical simulations in the infinite continuous classes regime.
Figure 6: Matrix of the overlaps between the first 100 eigenvectors of matrices Σxx and Σx∣c empirically estimated from COCO dataset embedded in a latent space via CLIP.
Figure 7: Left: speciation time ts as a function of the variance of the target distribution for multiple amounts of centroids in the target Gaussian mixture. For each value of β there exists one σo2 such that ts=0 . Center: onset variance σo2 in log-scale as a function of β . The more centroids there are, the lower is σo2 , i.e. the easier is for the centroids not to be local maxima of the Gaussian mixture. Right: critical entropy of centroids βc as a function of σ2 . The larger is σ2 , the lower is βc , i.e. the easier is for the centroids not to be local maxima of the Gaussian mixture.
Figure 8: Distortion observables for a choice of σ2,β that do not allow for the transition from the guided to the conditional phases of the potential. The system ends its run inside the guided phase. The color-map for the curves indicates different values of the guidance level: the darkest curve indicates w=0 , the lightest one w=1 .
Figure 9: Heat-maps representing the distortion at t=0 for various combinations of σ2,β,w when the target distribution is a Gaussian Mixture with an exponential mode count. The blank region in the speciation time panel indicates ts<0 , i.e. no transition from the guided to the conditional phase.
Figure 10: Measure of the speciation time ts and the distortion estimators in the exponential regime as a functions of the control parameters σ2 and w , for β=0.1 at sampling time t=0 . The white region in the ts panels signal that a negative speciation time, or equivalently, the absence of transition to the conditional phase. While this aspect is correlated to a stronger deformation of the target distribution, the occurrence of such transition appears to imply with a weak deformation, as testified by the behavior of δμ and δσ2 .
Figure 11: The effect of linear early-high CFG schedules where the guidance level is allowed to become negative on a multivariate Gaussian target, depending on two parameters w0 and ω . Panel (a) represents a phase diagram reporting the beneficial region in yellow: here both distortion weights are larget than unity. The outer region contains schedules where the mean is expanded and covariances are contracted. Panel (b) reports distortion estimators at t=0 measured from numerical simulations at different values of w0 and ω , for d1=1 and d2=9 . Specifically: the norm of the mean divided by the norm of the conditional mean on the left; the norm of the covariance matrix divided by the norm of the conditional covariance matrix on the right. As we can notice, for small values of w0 and small values of ω (i.e. a large negative guidance time window) we reach the yellow region of the phase diagram. The general effect is mild with this target distribution, because positive distortion is enhanced only along directions where the difference ri−si is large.
Figure 12: Distortion weights λ and Λ as functions of the eigenvalues (r,s)∈[0,1]2 , with s≤r , for a linear ealry-high CFG schedule with w0=−0.75 and ω=1 . The values are plotted in log-scale for enhancement. Panel (a) shows that the mean is expanded significantly only when si≃ri . Panel (b) displays a significant distortion of covariances in the opposite case, i.e. ri−si large. This behavior implies that the effects of such CFG prescription must be mild in general.
Figure 13: The effect of linear late-high CFG schedule where the guidance level is allowed to become negative, depending on two parameters w0 and ω . The variance of the mode is σ2=0.75 and the mode density β has been chosen to be large, so that the system never enters the conditional phase in the displayed domain. The first panel from the left is a phase diagram reporting the beneficial region in orange: here both means and variances are enlarged. The other regions either show loss of separability or tendency to shrink the mode variance. The following panels report distortion estimators. Dashed lines signal the passage from a negative distortion at time t=0 , to a positive one.
Figure 14: Evaluation of class separation metrics in feature space as a function of the guidance level w averaged over a dataset from Stable Diffusion (v1.5). Points are averaged over 50 prompts and 20 samples per prompt, errors are standard deviations of the mean. Blue circles refer to CLIP feature extractor, yellow ones to DINOv2.
Figure 15: Evaluation of data dispersion metrics in feature space as a function of the guidance level w averaged over a dataset from Stable Diffusion (v1.5). Points are averaged over 50 prompts and 20 samples per prompt, errors are standard deviations of the mean. Blue circles refer to CLIP feature extractor, yellow ones to DINOv2.
Figure 16: LPIPS as a function of the guidance level w averaged over a dataset from Stable Diffusion (v1.5). Points are averaged over 50 prompts and 20 samples per prompt, errors are standard deviations of the mean.
Figure 17: Samples of images generated with Stable Diffusion v1.5 at different guidance levels. Rows are different random seeds, columns refer to guidance levels. Prompts read, respectively: a photo of a red lighthouse on a rocky coast, dramatic sky, 35mm film , and a vibrant coral reef teeming with colorful fish, underwater scene, bright lighting .
Figure 18: Evaluation of different pairs of metrics for mean and variance distortion on several guidance schedules, features extracted with encoder CLIP. The black line corresponds to the fitted curve for a fixed guidance schedule. Round marks correspond to vanilla CFG schedules, polygons to negative window schedule, triangles to early-high schedules, star to early-low. The markers are colored according to the reference guidance levels reported in Fig. 4 .
Figure 19: Evaluation of different pairs of metrics for mean and variance distortion on several guidance schedules, features extracted with encoder DINOv2. The black line corresponds to the fitted curve for a fixed guidance schedule. Round marks correspond to vanilla CFG schedules, polygons to negative window schedule, triangles to early-high schedules, star to early-low. The markers are colored according to the reference guidance levels reported in Fig. 4 .
Figure 20: CLIP scores across several standard CFG schedules, quantifying the alignment of the data with the prompt representation. As predicted by the theory, a large gain in diversity, observed for larger negative windows, sacrifices the mean shift, i.e. prompt-alignment.
Figure 21: Comparison between images generated at w=0 (conditional), fixed w=3 (vanilla CFG), and negative-window guidance with different window widths, measured in diffusion time percentage. Intermediate windows allow to increase prompt alignment and the diversity of the samples. Prompt: a snowy mountain peak under a starry sky, northern lights, long exposure .
Figure 22: Comparison between images generated at w=0 (conditional), fixed w=6.5 (vanilla CFG), and negative-window guidance with different window widths, measured in diffusion time percentage. Intermediate windows allow to increase prompt alignment and the diversity of the samples. Prompt: a fantasy landscape with castles and dragons, vibrant colors, digital art .
Classifier-free guidance (CFG) is the default mechanism for conditional generation in diffusion models, but the distribution sampled by its deterministic guided dynamics is not captured by the usual product-distribution heuristic p0ωq01−ω. We analyze CFG through the probability flow ODE and derive exact analytic path-integral representations of the induced distributions for both constant and time-dependent guidance. The resulting formulas show that CFG modifies pt0 by an exponential path-integral correction, and that a time-dependent schedule enters this correction through the weight ω(t)−1. This characterization explains how score discrepancies accumulate along sampling trajectories and motivates Distribution-Guided CFG (DG-CFG), a schedule that balances timestep contributions while accounting for signal strength and low-noise score-error amplification. A toy model with analytic scores closely verifies the predicted distributions. Across Stable Diffusion1.5, Stable Diffusion2.1, and Stable DiffusionXL, DG-CFG yields a stronger diversity--fidelity trade-off and robustly mitigates the saturation and quality degradation caused by strong constant or heuristic guidance. Complete NFE experiments on Stable Diffusion1.5 and Stable Diffusion~2.1 confirm that these gains persist across sampling budgets, while fixed-quality experiments on both backbones show that DG-CFG reaches target metrics with fewer sampling steps.
Enze Jiang, Zheng Ma
School of Mathematical Sciences Shanghai Jiao Tong University Shanghai, China
Diffusion models generate samples by learning to reverse a fixed corruption process, and classifier-free guidance (CFG) is the standard mechanism for conditioning this process on a desired class or prompt. CFG can be applied at varying guidance strengths, and while higher strengths improve image quality and conditional alignment, too high a guidance strength can degrade image quality and diversity. Furthermore, CFG violates principled diffusion sampling dynamics, and existing explanations for why it works despite the violation disagree on the underlying theory or do not extend to deterministic samplers used in practice. We address both these issues. We first frame CFG sampling as a continuous-time optimal control problem, treating the sampling trajectory as a sequence of controls chosen to maximise the probability of the desired condition. Solving the resulting Hamilton--Jacobi--Bellman equation shows that CFG is recovered under specific path costs when using an unconstrained control set. We argue this lack of constraint is responsible for CFG's failure at high guidance strengths, since it permits the sampling path to move arbitrarily far from the current image estimate. To fix this, we propose curvature-aware CFG (CACFG), which constrains the control set to a hypersphere informed by the Gaussian regularisation used when training variational autoencoders. We show that the control inputs produced by CFG sampling routinely violate this bound, and that across diffusion models, datasets, and guidance schedules, CACFG achieves superior generative quality at mid-to-high guidance strengths with a less severe quality-diversity tradeoff than regular CFG.
Max Collins, Dasith de Silva Edirimuni, Jordan Vice +2
School of Physics, Maths and Computing The University of Western Australia Perth, WA 6009
Classifier-free guidance (CFG) is the standard way to strengthen class-conditioning in diffusion and flow-matching samplers, yet at large guidance it oversaturates and destabilizes, symptoms practitioners suppress with more steps or limited-interval schedules. We analyze CFG through an asymptotic-preserving, numerical-analysis lens. Building on a recent result that the deterministic DDIM step is the unique fitted operator for the unguided terminal layer, exact on the final small-sigma stretch of sampling, we show that guidance re-stiffens exactly the discriminative subspace to an anomalous exponent 1+w. DDIM is therefore no longer fitted there, and on coarse meshes its guided residual diverges as sigma_min goes to zero. We prove a guided clock barrier with three ordered step-size thresholds, and read one-step oversaturation as its endpoint: a solver artifact on the calibration model rather than the continuous guided law. The same analysis yields a one-coefficient, zero-extra-NFE repair: replace CFG's w(r-1) by r^(1+w)-r on the guidance direction. On the calibration model's discriminative crossover, this removes CFG's sigma_min-divergent blow-up and is first-order accurate against the exact guided flow as sigma_min goes to zero. On learned CIFAR-10 checkpoints, and as a cross-domain smoke test on Stable Diffusion 1.5 DDIM, it acts as a high-guidance stabilizer at no extra cost rather than a universal quality knob: it cuts residual amplification and saturation, gives 9/9 point-FID wins over CFG on the tested grid, and preserves classifier-proxy target accuracy in the hard-cell blocks. We report the limits alongside: it is not a universal image-quality win, and against a dense vanilla-CFG reference it is not a uniformly better integrator of that field.