We Have Been Interpreting Diffusion Transformer Backbone Behaviors Wrong

Benyuan Meng; Longtao Huang; Yang Liu; Liang Lin

cs.CVOct 10, 2026

We Have Been Interpreting Diffusion Transformer Backbone Behaviors Wrong

Authors: Benyuan Meng, Longtao Huang, Yang Liu, Liang Lin

Organizations: 1Sun Yat-sen University · 2Alibaba Group · 4X-Era AI Lab · 3Peng Cheng Laboratory

Abstract

During a network forward call, diffusion backbones naturally produce many intermediate activations. If we evaluate each activation against an identical downstream benchmarking protocol, we can plot an inverted-U-shaped curve as the depth profile: the activation's performance goes up first and then down as we go into deeper layers. This is common knowledge in the diffusion model community, which inspires many applications including the widely-adopted REPA. However, the common interpretation of this depth profile may be inaccurate and incomplete: (i) The cause of this curve has long been treated as signal granularity changing from coarse to fine, but there has been little evidence in DiT backbones. (ii) The community has been mainly studying this phenomenon in a static way, without investigating how it changes across different training setups. In this work, we show a different mechanism picture for the profile, where the prediction target (e.g., $ε$ , $u$ , $x_0$ ) sits at the center. The semantically meaningful signal is only an intermediate by-product for predicting the target, while the prediction target is the real goal for the backbone. Furthermore, the choice of prediction targets as part of the training setup can predictably affect the curve's shape. With this updated interpretation, many applications of this depth profile can be better grounded and predicted. As an example, we show how to predict the near-optimal placement of REPA based on the prediction target of the training setup. The code is at https://github.com/Darkbblue/dit-compare.

Figures & tables

Figure 2: (a) A visualization of the default min-SNR weighting (

\gamma=5

) in the

u

-space. (b) A typical depth profile of diffusion backbones. The curve is obtained from FM+

u

model with clean weighting.

Figure 3: (a) Probing the same FM+ $u$ model at different SNR yields highly similar depth profiles. (b) Basic training-free correspondence probe results on different sampling paths. (c) Whitening correspondence probe results on different sampling paths.

Figure 4: (a) An illustration of what each phase writes into the main residual and remains there afterward . (b) Geometry probe results of FM+ $a45$ model, angle measured in cosine. Each feature is measured relative to the layer input.

Figure 5: (a) Training-free correspondence probe results from the original and the modulated FM+ $a45$ models. (b) Separability of FM+ $a45$ model’s MLP increments, measured using the remaining $x_{0}$ signal after deleting $x_{0}$ from the feature while minimizing the damage to $\epsilon$ .

Figure 6: (a) Linear regression probe results of $x_{0}$ , $x_{s}$ via peak feature proxy, and $x_{s}$ via DINOv2 proxy. (b) Correspondence probe results from the FM+ $\epsilon$ (clean weighting) models, using the baseline training-free probe, linear convolution probe, whitening probe, and transplantation probe (projecting features into the peak feature’s space).

Figure 7: (a) Models trained using the basic min-SNR weighting (emphasizing cleaner samples) with different prediction targets. (b) Models trained using the neutral weighting with different prediction targets. (c) $x_{0}$ - and $\epsilon$ -models trained with three loss weighting methods.

Setting	$log$ SNR=1.69				$log$ SNR=-1.69
Setting	seed 1	seed 2	seed 3	avg	seed 1	seed 2	seed 3	avg
$x_{0}$ (original)	0.0741	0.0720	0.0734	0.0732	0.3352	0.3347	0.3344	0.3348
$\epsilon$ (original)	0.3969	0.3969	0.3941	0.3960	0.0618	0.0642	0.0618	0.0626
$x_{0}$ (replacing $x_{0}$ )	0.1767	0.1833	0.1816	0.1805	0.8191	0.8599	0.8336	0.8375
$x_{0}$ (replacing $\epsilon$ )	0.1838	0.1813	0.1841	0.1831	0.7542	0.7758	0.7828	0.7709
$\epsilon$ (replacing $x_{0}$ )	0.6127	0.6162	0.6274	0.6188	0.2258	0.2323	0.2314	0.2298

Table 1: Causal intervention by replacing a certain component within the MLP increments with random noise. The more severe loss increase is marked as bold . The same seed is shared across the same column that controls both image sampling and noise sampling. The absolute loss value is not comparable across models or SNR. All interventions are done with FM+

x_{0}

and FM+

\epsilon

models trained under the neutral weighting.

REPA Layer	5	7	9	11	13	Dynamic	Inverted Dynamic
FID $\downarrow$	15.56	13.4	12.49	12.31	12.33	11.43	12.33
IS $\uparrow$	101.96	116.6	124.56	125.63	123.6	133.2	124.34

Table 2: Generation performance of aligning REPA at different layers during training. Best marked as bold and second marked as underlined . The models are trained for 100k steps, and the evaluation is done on 10k samples generated using CFG=1.5.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: (a) Models of different prediction targets trained with a fixed VAE scaling factor, under the neutral loss weighting. (b) Models trained under different loss weighting with a fixed VAE scaling factor. Models in both figures are trained for 75k steps.

Figure 9: Using separability probe to measure the length of the preparation phase as the training goes on. We can observe that the preparation phase gradually shortens.

Figure 10: Using separability probe to depict the roughly symmetric pattern caused by prediction targets. The models are trained with the clean weighting.

Figure 11: (a) Using separability probe to measure the REPA-augmented model, finding that the separability in early layers significantly improves. (b) The correspondence performance as well as the curve shape are also changed greatly.

Figure 12: Measuring the separability of the same FM+ $a15$ model with whitening and with different SNR. This result shows the robustness of separability probe as the definition of the length of the preparation phase.

Model	$x_{0}$ -clean	$x_{0}$ -neutral	$x_{0}$ -noisy	$\epsilon$ -clean	$\epsilon$ -neutral	$\epsilon$ -noisy
FID $\downarrow$	62.66	60.55	72.16	61.62	63.22	82.71
IS $\uparrow$	50.62	59.61	41.16	55.19	52.81	31.80

Appendix

Table 3: We measure the generation performance of well-conditioned and degenerated models. The results appear to suggest that degeneration does not necessarily lead to poor generation. All models use FM paths.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,192 papers

Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis
Jul 4, 2026Haksoo Lim, Myeongjin Lee, Wonjoon Chang +1Diffusion Models
Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea · INEEJI, Seongnam 13558, Republic of Korea
Scaffold Then Internalize: Representation Injection for Diffusion Transformers
Sep 28, 2026Han Fu, Jiacheng Chen, Baoquan Zhao +4Diffusion Transformer Diffusion Model Acceleration
Sun Yat-sen University · Video Rebirth · The Hong Kong Polytechnic University
Why DiT Models Underperform as Representation Learners without Long Skip Connections
Oct 9, 2026Benyuan Meng, Yiliang Zhang, Jin-Wen Wu +3Diffusion Models Diffusion Transformer
Institute of Information Engineering, CAS · School of Cyber Security, University of Chinese Academy of Sciences · Alibaba Group +4

Jul 4, 2026J/K move · Enter open · S save

Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis

Haksoo Lim, Myeongjin Lee, Wonjoon Chang, Jaesik Choi

Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea · INEEJI, Seongnam 13558, Republic of Korea

Diffusion Models

Open paper AskPDF arXiv

Diffusion models have achieved remarkable success across diverse domains, with performance closely related to the denoising backbones that parameterize the score function. In this paper, we present a systematic, phase-aware analysis of diffusion components and show that abrupt, early-stage fluctuations in deep latents are strongly associated with artifacts. Guided by these findings, we introduce DUNE (Diffusion Unified Network refiNEr), a training-free refinement framework that detects abrupt deviations in deep low-noise internal latents using a shared EMA-based criterion, and applies backbone-specific suppression to the detector-selected entries. Although derived from U-Net, the same detect-suppress principle extends naturally to Transformer-based diffusion models by acting on the latents of deep self-attention blocks. Extensive experiments across multiple backbones indicate that DUNE improves fidelity while reducing hallucinations, offering new insight into where and when diffusion backbones should be controlled.

Abstract

Diffusion Models

Explore similar work

Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis

Scaffold Then Internalize: Representation Injection for Diffusion Transformers

Why DiT Models Underperform as Representation Learners without Long Skip Connections