Organizations: Leinweber Institute for Theoretical Physics and Department of Physics, University of California, Berkeley, CA, 94720-7300, USA · Theoretical Physics Group, Lawrence Berkeley National Laboratory, Berkeley, CA 94720-8162, USA · Department of Radiation Oncology, University of California, San Francisco · Computational Precision Health, University of California, San Francisco · Department of Physics, University of California, Berkeley, CA 94720, USA
Recent theoretical models of diffusion processes, conceptualized as coupled Ornstein-Uhlenbeck systems, predict a hierarchy of interaction timescales, and consequently, the existence of a synchronization gap between modes that commit at different stages of the reverse process. However, because these predictions rely on continuous time and analytically tractable score functions, it remains unclear how this phenomenology manifests in the deep, discrete architectures deployed in practice. In this work, we investigate how the synchronization gap is mechanistically realized within pretrained Diffusion Transformers (DiTs). We construct an explicit architectural realization of replica coupling by embedding two generative trajectories into a joint token sequence, modulated by a symmetric cross attention gate with variable coupling strength g. Through a linearized analysis of the attention difference, we show that the replica interaction decomposes mechanistically. We empirically validate our theoretical framework on a pretrained DiT-XL/2 model by tracking commitment and per layer internal mode energies. Our results reveal that: (1) the synchronization gap is an intrinsic architectural property of DiTs that persists even when external coupling is turned off; (2) as predicted by our spatial routing bounds, the gap completely collapses under strong coupling; (3) the gap is strictly depth localized, emerging sharply only within the final layers of the Transformer; and (4) global, low frequency structures consistently commit before local, high frequency details. Ultimately, our findings provide a mechanistic interpretation of how Diffusion Transformers resolve generative ambiguity, isolating speciation transitions to the terminal layers of the network.
Figures & tables
Figure 1: For coupling strength g=0 and speciation step s=53 , we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). Even with coupling turned off, the gap is present at deep layers of DiT.
Figure 2: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Figure 3: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Figure 4: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Coupling strength g
τspec(g) [95% CI]
0.1
63.6[50.0,96.6]
0.3
48.0[33.7,85.3]
0.5
46.3[31.6,82.6]
0.7
45.6[35.0,59.7]
0.9
45.0[35.8,56.0]
1.0
44.2[35.6,58.8]
Table 1: Summary of speciation times determined using cosine similarity method and corresponding confidence intervals vs coupling strength g .
Figure 5: A separate analysis of the speciation time using the median feature space cosine similarity ( 70 ) between paired outputs for the vanishing coupling strength g=0 . In this modified setup, the noise of reverse process for both replicas are shared until intervention time when it becomes independent.
Figure 6: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 7: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 8: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 9: For particular coupling strength g and speciation time we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). These sweeps reveal that the gap is located at deep layers of the Transformer and as coupling strength g increases the gap collapses.
Figure 10: For particular coupling strength g and speciation time, we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). These sweeps reveal that the gap is located at deep layers of the Transformer and as coupling strength g increases the gap collapses.