Organizations: Leinweber Institute for Theoretical Physics and Department of Physics, University of California, Berkeley, CA, 94720-7300, USA · Theoretical Physics Group, Lawrence Berkeley National Laboratory, Berkeley, CA 94720-8162, USA · Department of Radiation Oncology, University of California, San Francisco · Computational Precision Health, University of California, San Francisco · Department of Physics, University of California, Berkeley, CA 94720, USA
Recent theoretical models of diffusion processes, conceptualized as coupled Ornstein-Uhlenbeck systems, predict a hierarchy of interaction timescales, and consequently, the existence of a synchronization gap between modes that commit at different stages of the reverse process. However, because these predictions rely on continuous time and analytically tractable score functions, it remains unclear how this phenomenology manifests in the deep, discrete architectures deployed in practice. In this work, we investigate how the synchronization gap is mechanistically realized within pretrained Diffusion Transformers (DiTs). We construct an explicit architectural realization of replica coupling by embedding two generative trajectories into a joint token sequence, modulated by a symmetric cross attention gate with variable coupling strength g. Through a linearized analysis of the attention difference, we show that the replica interaction decomposes mechanistically. We empirically validate our theoretical framework on a pretrained DiT-XL/2 model by tracking commitment and per layer internal mode energies. Our results reveal that: (1) the synchronization gap is an intrinsic architectural property of DiTs that persists even when external coupling is turned off; (2) as predicted by our spatial routing bounds, the gap completely collapses under strong coupling; (3) the gap is strictly depth localized, emerging sharply only within the final layers of the Transformer; and (4) global, low frequency structures consistently commit before local, high frequency details. Ultimately, our findings provide a mechanistic interpretation of how Diffusion Transformers resolve generative ambiguity, isolating speciation transitions to the terminal layers of the network.
Figures & tables
Figure 1: For coupling strength g=0 and speciation step s=53 , we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). Even with coupling turned off, the gap is present at deep layers of DiT.
Figure 2: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Figure 3: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Figure 4: Each plot shows the median feature space cosine similarity ( 70 ) between paired outputs as a function of intervention step tint , together with the sigmoid fit used to extract τspec(g) . With an increase of coupling strength g , the speciation time decreases.
Coupling strength g
τspec(g) [95% CI]
0.1
63.6[50.0,96.6]
0.3
48.0[33.7,85.3]
0.5
46.3[31.6,82.6]
0.7
45.6[35.0,59.7]
0.9
45.0[35.8,56.0]
1.0
44.2[35.6,58.8]
Table 1: Summary of speciation times determined using cosine similarity method and corresponding confidence intervals vs coupling strength g .
Figure 5: A separate analysis of the speciation time using the median feature space cosine similarity ( 70 ) between paired outputs for the vanishing coupling strength g=0 . In this modified setup, the noise of reverse process for both replicas are shared until intervention time when it becomes independent.
Figure 6: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 7: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 8: Each plot shows a global ( 71 ) and local ( 72 ) commitment scores vs coupling strength g . These show that coarse output structure stabilizes substantially earlier than fine detail throughout the explored coupling range.
Figure 9: For particular coupling strength g and speciation time we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). These sweeps reveal that the gap is located at deep layers of the Transformer and as coupling strength g increases the gap collapses.
Figure 10: For particular coupling strength g and speciation time, we sweep across all Transformer layers to evaluate the normalized fixed basis energies of leading and trailing internal difference modes ( 78 )–( 79 ). These sweeps reveal that the gap is located at deep layers of the Transformer and as coupling strength g increases the gap collapses.
Diffusion Transformers (DiTs) have established themselves as a scalable backbone for high-fidelity image synthesis. However, unlike U-Net based diffusion models that rely on rigid, hand-crafted skip connections, DiTs predominantly use a uniform residual stream that integrates all preceding layers as a monolithic state. In this work, we rethink residual connections in diffusion transformers and propose to transform them from passive summation into an active retrieval mechanism optimized for image denoising. First, we conduct a systematic analysis of DiT's internal representation, revealing a latent preference for early-layer feature reuse and symmetric layer guidance. Motivated by this, we introduce a structured connectivity design that explicitly integrates local residual connections with long-range pathways. Instead of static skip connections or dense all-layer routing, our method enables each transformer block to selectively ``attend'' to critical earlier representations, dynamically retrieving spatial and semantic cues through direct, differentiable cross-depth paths. Experiments show that our adaptive connectivity leads to faster convergence, with up to 1.73× fewer training iterations, and significant gains in FID and visual quality with less than 0.1% additional parameters, further improving a strong REPA-XL/2 model from 5.9 to 4.34 FID without guidance and reaching 1.39 FID with classifier-free guidance. Our findings suggest that adaptive cross-layer connectivity is a critical yet underexplored factor in diffusion transformers, and that incorporating structured information pathways provides a simple and effective direction for improving scalable generative models.
Yuhe Liu, Xinyin Ma, Gongfan Fang +2
National University of Singapore · Shanghai Jiao Tong University
We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.
Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García +2
Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)
Recent breakthroughs in Diffusion Transformers (DiTs) have revolutionized the field of visual synthesis due to their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs are not well understood. To this end, we first systematically investigate the representation dynamics of DiTs. Through analyzing the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a crucial factor for effective learning. Based on this key insight, we propose DiverseDiT, a novel framework that explicitly promotes representation diversity. DiverseDiT incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet 256x256 and 512x512 demonstrate that our DiverseDiT yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes, even when tested on the challenging one-step generation setting. Furthermore, we show that DiverseDiT is complementary to existing representation learning techniques, leading to further performance gains. Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance.
Mengping Yang, Zhiyu Tan, Binglei Li +3
Fudan University · Shanghai Academy of AI for Science · Shanghai Innovation Institute