Breakdown of Local Denoising as Semantic Speciation
Authors: Guangkuo Liu, Mert Okyay, Yifan F. Zhang, Fangjun Hu, Rahul Nandkishore, Xun Gao
Organizations: JILA and Department of Physics University of Colorado Boulder Boulder, CO 80309, USA · CTQM and Department of Physics University of Colorado Boulder Boulder, CO 80309, USA · Department of Electrical and Computer Engineering Princeton University Princeton, NJ 08544, USA · QuEra Computing Inc. 1284 Soldiers Field Road, Boston, MA 02135, USA
The dynamics of generative models exhibit two apparently distinct temporal windows: a speciation window, in which a sample commits to a semantic class, and a nonlocality window, in which local context windows become insufficient for generation. Motivated by evidence of their near-concurrence in a variety of frontier models, we investigate their relationship through the spatial distribution of semantic information. Under a "common cause" hypothesis, we prove that the nonlocality window must lie in the speciation window. This hypothesis postulates that semantic labels explain a fraction of the correlations between distant tokens, a condition that is natural for many real datasets. We further give conditions under which both windows shrink to a single limiting time as system size grows, defining a "phase transition", and verify this behavior analytically in Gaussian mixtures. Together, these results identify conditions under which semantic information explains the concurrence of speciation and nonlocality, connecting two complementary perspectives on the emergence of semantic structure in generative modeling.
Figures & tables
Figure 1: Semantic speciation locates the window in which local denoising breaks down. Denoising trajectory is from t=1 (right) to t=0 (left). The upper panel depicts a semantic speciation from white noise to cat or dog for local region (blue) and global region (magenta); the corresponding thresholds define a speciation window (black vertical lines). The noisy images in the window depict generation times in which the semantic identity is visible in the whole image, but invisible in a local patch (blue box). In the lower row, A is the patch to denoise (green), B its minimal surrounding context (blue) for below-threshold denoising, and C the rest of the image (magenta). The green curve depicts It(A;C∣B) whose peak characterizes a breakdown of exact denoising using local context alone. Under the common-cause hypothesis, Theorem 1 places the threshold-defined nonlocality window inside the speciation window.
Figure 2: Semantic conditioning reduces the average local–global prediction gap in SD3. A smaller conditional gap means that restricting distant image-token interactions changes the model prediction less when semantic information is supplied. This is consistent with the common-cause interpretation: the description explains part of the shared image structure that would otherwise require distant context.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Label–reconstruction similarity remains high at early diffusion times and then falls toward zero. The sharp drop occurs first for the 64×64 crop, then for the 128×128 crop, and last for the full image.
Figure 4: Exact numerical evaluation of the Gaussian-mixture information measures with specific values given in E.3 . (a) Mutual information curves for the four spatial regions. The cliff diagnoses the symmetry-breaking phase transition. Inset: zoomed-in region near critical window. (b) Local and global conditional information gaps It(S;A∣B),It(S;A∣BC) , and their difference which gives conditional interaction information It(A;C;S∣B) . The peak diagnoses the non-locality transition.
Diffusion models undergo a phase transition in a critical time window during generation dynamics, with two complementary diagnoses of criticality. The symmetry breaking picture views the critical window as when trajectories bifurcate into different semantic minima of the energy landscape, whereas the nonlocality picture views the critical window as when local denoising fails. We study whether two notions of such phase transitions are concurrent in modern diffusion transformers. By evaluating the dynamics and outcomes of the generation trajectory, we observe a near-simultaneous occurrence of the non-locality and symmetry breaking critical times. Our work is the first to unify the two notions of phase transitions in practice: it provides a concrete diagnostic for when and why diffusion models rely on conditioning and global denoising, enabling principled evaluation of model efficiency and guiding the design of architectures and sampling schemes that avoid unnecessary computation.
Yifan F. Zhang, Fangjun Hu, Guangkuo Liu +2
Department of Electrical and Computer Engineering Princeton University Princeton, NJ 08544, USA · QuEra Computing Inc. 1284 Soldiers Field Road, Boston, MA 02135, USA · JILA and Department of Physics University of Colorado Boulder Boulder, CO 80309, USA +1
Speciation in generative diffusion models denotes the emergence of distinct stable branches during denoising, through which initially undifferentiated trajectories progressively commit to different data classes. In this work we develop an intrinsic theory of speciation for diffusion models supported on compact Riemannian manifolds: the aim is to go beyond existing theoretical descriptions, which usually identify speciation with a symmetric pitchfork bifurcation and assume to work in a large-dimensional space. We characterize speciation by bifurcations of the critical points of the evolving probability density. A spectral heat-kernel representation makes explicit the role of the manifold geometry, while Poincaré-Hopf and Morse theory impose global constraints on the number and type of score equilibria and reveal topologically-imposed geometrical modes. For mixtures of heat kernels, we prove that generic speciation events have a one-dimensional critical kernel and admit an A2 fold normal form; pitchforks and simultaneous multidirectional transitions arise from nongeneric symmetric configurations. We derive geometry-dependent estimates of speciation times for bimodal mixtures and Riemannian regular simplices. We further establish structural stability of nondegenerate folds under score perturbations and show that the first-order time shift is determined solely by the component of the score error along the critical direction. The theory is illustrated on the sphere using mixtures of von Mises-Fisher distributions, where pitchfork and saddle-node bifurcations, topological modes, and hierarchical multiple speciations are observed. Finally, a chart-based intrinsic score-learning scheme based on neural networks contrasts the theoretically predicted transitions on prototypal and more complex datasets.
Alessio Marta, Paola Causin
Department of Mathematics, University of Milan, via Saldini 50, 20133 Milan, Italy
We show that deliberately breaking detailed balance in generative diffusion processes can accelerate the reverse process without changing the stationary distribution. Considering the Ornstein--Uhlenbeck process, we decompose the dynamics into a symmetric component and a non-reversible anti-symmetric component that generates rotational probability currents. We then construct an exponentially optimal non-reversible perturbation that improves the long-time relaxation rate while preserving the stationary target. We analyze how such non-reversible control reshapes the macroscopic dynamical regimes of the phase transitions recently identified in generative diffusion models. We derive a general criterion for the speciation time and show that suitable non-reversible perturbations can accelerate speciation. In contrast, the collapse transition is governed by a trace-controlled phase-space contraction mechanism that is fixed by the symmetric component, and the corresponding collapse time remains unchanged under anti-symmetric perturbations. Numerical experiments on Gaussian mixture models support these findings.
Haiqi Lu, Ying Tang
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · Yingcai Honor College, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Physics, University of Electronic Science and Technology of China, Chengdu 611731, China +2