Unlocking the Potential of Contrastive Regularization for Diffusion Transformer Training
Organizations: NTU · USTC · Tongji University · HFUT ·
Abstract
Representation regularization has emerged as an effective way to improve diffusion transformer training. Yet one of the most canonical regularizers--InfoNCE-style contrastive learning--remains largely underexplored in this setting. We naturally ask a question: can standard contrastive learning serve as an effective regularizer for diffusion transformers? Surprisingly, we find that naive InfoNCE regularization brings only marginal gains. Our analysis reveals that the objective is easily dominated by a shortcut: near-orthogonal noise components in diffusion inputs provide a bypass for enlarging negative-pair distances while weakening positive-pair alignment, especially at high noise levels. To unlock the benefit of contrastive regularization, we propose Clean Representation-ANchored Contrastive (CRAN) regularization. Specifically, CRAN anchors contrastive dispersion with clean-sample representation distances, transferring clean relational semantic structure to noisy diffusion representations to suppress the noise shortcut. Extensive experiments on ImageNet show that CRAN consistently improves diffusion transformer training across model scales and training budgets, outperforming internal feature-alignment and dispersive regularization baselines while matching external DINO-guided REPA. These results re-establish InfoNCE-based contrastive learning as an effective regularizer for diffusion training, once its noise-driven over-dispersion is properly constrained.
Figures & tables
| Method | Regularization | FID ( ) |
|---|---|---|
| Baseline | None | 33.0 |
| SRA | Positive only | 28.7 |
| InfoNCE | Positive + Negative | 28.5 |
| CRAN (ours) | Anchored InfoNCE | 26.3 |
| Type | Method | Target / Pair Construction | Regularization Objective |
| P.-only | REPA † ( Yu et al., 2024 ) | External pretrained representation | |
| LayerSync ( Haghighi et al., 2025 ) | Detached deeper-layer representation | ||
| SRA ( Jiang et al., 2026 ) | Deeper and less noisy EMA representation | ||
| SRA2 ( Wang et al., 2026 ) | Reconstructing VAE latent of clean image | ||
| N.-only | Disp. Loss ( Wang & He, 2025 ) | In-batch negatives | |
| P. + N. | InfoNCE | Positive and in-batch negatives |
| Method | dis. metric | Dis. Anchor | FID | sFID | IS | Pre. | Rec. | |
| SiT-B/2 | - | - | - | 33.0 | 6.46 | 43.7 | 0.53 | 0.63 |
| + CRAN | - | 1.0 | 31.9 | 6.53 | 44.8 | 0.55 | 0.63 | |
| + CRAN | - | 1.0 | 28.3 | 5.92 | 50.9 | 0.57 | 0.63 | |
| + CRAN | 1.0 | 27.9 | 5.99 | 51.3 | 0.57 | 0.64 | ||
| + CRAN | 1.0 | 27.0 | 6.06 | 53.9 | 0.58 | 0.64 | ||
| + CRAN | 1.0 | 28.6 | 5.99 | 50.5 | 0.57 | 0.64 |
| Method | Pos. | Neg. | FID |
|---|---|---|---|
| SiT-B/2 | No | No | 33.0 |
| CRAN | No | Yes | 31.9 |
| CRAN | Yes | No | 28.7 |
| CRAN | Yes | Yes | 26.3 |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Noise level | SiT | InfoNCE | CRAN |
|---|---|---|---|
| 0.10 | 0.8950 | 0.8232 | 0.8789 |
| 0.40 | 0.3638 | 0.3150 | 0.3417 |
| 0.80 | 0.0288 | 0.0277 | 0.028 |
| Model | Anchor | FID | |
|---|---|---|---|
| SiT-B/2 | – | – | 33.0 |
| + Pos. reg. | – | – | 28.7 |
| + InfoNCE | 10.0 | – | 28.3 |
| + InfoNCE | 5.0 | – | 28.6 |
| + InfoNCE | 1.0 | – | 28.5 |
| + InfoNCE | 0.5 | – | 28.9 |
| Model | iter. | FID | IS |
|---|---|---|---|
| SiT-B/2 | 400k | 33.0 | 43.7 |
| SiT-B/2 + CRAN | |||
| t=0.1 | 400k | 26.3 | 56.1 |
| t=0.5 | 400k | 26.7 | 55.6 |
| t=0.9 | 400k | 28.2 | 50.8 |
| Student | Teacher | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|
| 2 | 8 | 44.8 | 11.5 | 33.5 | 0.49 | 0.62 |
| 3 | 8 | 42.9 | 11.4 | 36.0 | 0.50 | 0.62 |
| 4 | 8 | 44.2 | 11.4 | 33.7 | 0.50 | 0.62 |
| 3 | 6 | 51.2 | 13.4 | 30.3 | 0.45 | 0.61 |
| 3 | 10 | 42.9 | 11.7 | 35.4 | 0.50 | 0.61 |
| Dataset | Method | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|
| FFHQ | SiT | 6.73 | 7.55 | 4.4 | 0.71 | 0.47 |
| FFHQ | REPA | 6.49 | 7.63 | 4.5 | 0.71 | 0.48 |
| FFHQ | CRAN | 5.59 | 7.02 | 4.6 | 0.71 | 0.49 |
| WebVision-10-50K | SiT | 27.6 | 9.84 | 15.2 | 0.45 | 0.47 |
| WebVision-10-50K | CRAN | 20.5 | 9.45 | 15.8 | 0.48 | 0.47 |
| Method | Steps | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|
| SiT | 200K | 56.7 | 8.11 | 26.8 | 0.53 | 0.62 |
| CRAN | 200K | 45.5 | 6.98 | 32.7 | 0.61 | 0.62 |
| SiT | 400K | 43.3 | 7.41 | 36.8 | 0.61 | 0.64 |
| CRAN | 400K | 32.9 | 6.57 | 47.4 | 0.67 | 0.63 |
| Method | Steps | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|
| DiT-B/2 | 400K | 43.5 | 8.71 | 32.7 | 0.49 | 0.63 |
| DiT-B/2 + CRAN | 400K | 38.6 | 8.24 | 37.6 | 0.51 | 0.64 |
| Method | FID | R-Precision |
|---|---|---|
| MDM | ||
| MDM + REPA | ||
| MDM + CRAN |
| Backbone | Method | Time/step | Ratio | GFLOPs/sample | Peak memory |
|---|---|---|---|---|---|
| B/2 | SiT | 36.92 ms | 1.000 | 139.2 | 6.29 GiB |
| B/2 | SRA | 41.19 ms | 1.116 | 173.5 | 6.41 GiB |
| B/2 | CRAN | 41.32 ms | 1.119 | 173.5 | 6.45 GiB |
| B/2 | REPA | 52.39 ms | 1.419 | 194.4 | 6.92 GiB |
| L/2 | SiT | 99.01 ms | 1.000 | 487.4 | 18.66 GiB |
| L/2 | SRA | 108.56 ms | 1.096 | 601.4 | 18.83 GiB |
| Budget | Method | Steps | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|---|
| SiT | SiT | 448K | 33.7 | 6.59 | 44.4 | 0.54 | 0.64 |
| SiT | CRAN | 400K | 26.3 | 6.16 | 56.2 | 0.58 | 0.63 |
| REPA | REPA | 400K | 24.4 | 6.40 | 59.9 | 0.59 | 0.65 |
| REPA | CRAN | 507K | 24.5 | 6.12 | 59.6 | 0.59 | 0.64 |
| Method | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|
| REPA | 1.80 | 4.50 | 284.0 | 0.81 | 0.61 |
| SRA | 1.89 | 4.47 | 278.7 | 0.81 | 0.61 |
| CRAN | 1.83 | 4.35 | 286.6 | 0.83 | 0.59 |
| Method | Interval | CFG | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|---|
| CRAN | 1.70 | 1.52 | 4.76 | 277.4 | 0.80 | 0.62 | |
| CRAN | 1.80 | 1.49 | 4.78 | 289.6 | 0.81 | 0.62 | |
| CRAN | 1.90 | 1.54 | 4.83 | 299.5 | 0.81 | 0.61 | |
| CRAN | 1.70 | 1.44 | 4.75 | 295.5 | 0.81 | 0.64 | |
| CRAN | 1.80 | 1.45 | 4.68 | 306.6 | 0.81 | 0.63 | |
| CRAN | 1.90 | 1.64 | 4.71 | 323.9 | 0.82 | 0.61 |
| Steps | Method | FID | sFID | IS | Pre. | Rec. |
|---|---|---|---|---|---|---|
| 100K | REPA | 49.5 | 7.00 | 27.5 | 0.46 | 0.59 |
| 100K | REPA + CRAN | 42.4 | 6.64 | 32.5 | 0.50 | 0.60 |
| 200K | REPA | 33.2 | 6.68 | 43.7 | 0.54 | 0.63 |
| 200K | REPA + CRAN | 28.5 | 6.87 | 51.4 | 0.57 | 0.63 |
| 400K | REPA | 24.4 | 6.40 | 59.9 | 0.59 | 0.65 |
| 400K | REPA + CRAN | 21.0 | 6.46 | 68.6 | 0.60 | 0.65 |
| Model | #Params | Iter. | FID |
|---|---|---|---|
| SiT-B/2 | 130M | 400k | 33.0 |
| + SRA | 130M | 400k | 29.1 |
| + SRA & Disp. | 130M | 400k | 28.6 |
| + DiverseDiT | 130M | 400k | 28.1 |
| + SRA2 | 130M | 400k | 28.9 |
| + CRAN , full | 130M | 400k | 26.2 |
| computation | Time/step | Relative time |
|---|---|---|
| Full | 43.20 ms | 1.000 |
| Estimated | 41.32 ms | 0.956 |
| Backbone | SiT-B | SiT-L | SiT-XL |
|---|---|---|---|
| Architecture | |||
| Input | |||
| Patch size | 2 | 2 | 2 |
| Layers | 12 | 24 | 28 |
| Hidden dim. | 768 | 1,024 | 1,152 |
| Num. heads | 12 | 16 | 16 |
Contrastive Learning
Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.
See all 348 papers